P ROVABLE H IERARCHY -B ASED M ETA -R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors.
However, existing work either assume access to expert-constructed hierarchies, or use hierarchylearning heuristics with no provable guarantees.
To address this gap, we analyze
HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task.
We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics.
Analogous to supervised meta-learning theory, we provide “diversity conditions” which, together with a tractable optimism-based algorithm, guarantee sampleefﬁcient recovery of this natural hierarchy.
Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task.
Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of
HRL in practice.
I NTRODUCTION
Reinforcement learning (RL) has demonstrated tremendous successes in many domains (Schulman et al., 2015; Vinyals et al., 2019; Schrittwieser et al., 2020), learning near-optimal policies despite limited supervision.
Nevertheless, RL remains difﬁcult to apply to problems requiring temporally extended planning and/or exploration (Ecoffet et al., 2021).
A promising approach to this problem is hierarchical reinforcement learning (HRL), which has seen continued interest due to its appealing biological basis.
In its most basic form, HRL seeks to decompose tasks into a sequence of skills, each of which is easier to learn individually than the full task.
By restricting the agent to using learned skills, the search space over policies can be greatly reduced.
Furthermore, learned skills can induce simpler state and/or action spaces, simplifying the learning problem.
Finally, learned skills with useful semantic behavior can be reused across tasks, enabling transfer learning.
Naturally, a hierarchy-based learner is limited by the quality of skills that are made available and/or learned.
Accordingly, many empirical works have proposed algorithms for online skill learning in the context of a single RL task (Nachum et al., 2019a; 2018).
These approaches have been experimentally demonstrated to be effective in ﬁnding useful and interpretable skills.
Other approaches consider the skill learning problem in the context of meta-RL (Frans et al., 2018), or in the rewardfree setting (Eysenbach et al., 2018).
Nevertheless, the heuristics and algorithms proposed in these empirical works do not provide any provable guarantees on the quality of learned skills.
On the other hand, theoretical analyses have mostly focused on how learners beneﬁts from having access to skills.
For example, Fruit & Lazaric (2017) provide a regret bound on learning with skills in the inﬁnite-horizon average reward case.
Meanwhile, in the meta-RL setting, Brunskill & Li (2014) considers the problem of ﬁnding and using skills in a continual learning setting and provides a sample complexity analysis.
However, these analyses either sidestep the question of how the skills are obtained, or do not address the problem in a computationally tractable manner.
In this work, we aim to take a step towards providing provable guarantees for hierarchy learning through tractable algorithms.
We focus on the meta-RL setting, in which a learner extracts skills from a set of provided tasks which are then used in a downstream task.
We work in the tabular case, assuming the transition dynamics of the given tasks share latent hierarchical structure induced by predetermined clustering and bottlenecks.
Our contributions are as follows:
“Diversity conditions” ensuring hierarchy recovery.
We develop natural optimismbased coverage conditions which ensure that bottlenecks embedded in the transition dynamics are detectable by solving provided meta-training tasks.
A tractable hierarchy-learning algorithm.
We provide an algorithm that provably learns the latent hierarchy from interactions, assuming the coverage conditions above.
Our method has sample complexity scaling as O ( TKS ) in the leading term compared to O ( TS 2 A ) for a brute-force method, where T are the number of tasks, S is the number of states, A is the number of actions, and K (cid:28) SA is the number of skills to learn.
Regret bounds on downstream tasks.
We provide regret bounds for learners that apply the extracted hierarchy from meta-training on downstream tasks.
Furthermore, we show an exponential regret separation between hierarchy-based and hierarchy-oblivious learners for a family of task distributions, corroborating prevailing intuitions regarding when/why HRL helps.
In particular, hierarchy-based learners incur regret bounded by O ( √ H 2 N ) while hierarchy-oblivious learners incur worst-case regret of at least O (2 H/ 2 √ H 2 N ) .
N OTATION
We now introduce notation which we will use throughout the paper.
We write [ K ] := { 1 , . . .
, K } .
Furthermore, we use the standard notations O, Θ , Ω to denote orders of growth, and ˜ O, ˜Θ , ˜Ω to indicate suppressed logarithmic factors.
We use δ ( x ) to denote the Dirac delta measure on x .
We work with ﬁnite-horizon Markov decision processes (MDPs), deﬁned as a tuple M =
( S , A , P , r, H ) , where S is the set of states, A is the set of actions, P : S ×
A × S → [0 , 1] are the transition dynamics, r : S ×
A → [0 , 1] is the reward function, and H is the horizon.
We assume stationary dynamics unless otherwise noted, in which case P ( h ) is the dynamics at time step h .
For constants relating to horizons, we will deﬁne [ H ] := { 0 , . . .
, H − 1 } .
Given a policy π : [ H ] × S → A , we deﬁne the value functions V
[Equation]
and Q πh ( s, a ) :
= E (cid:34) H
[Equation]
[Equation]
k
[Equation]
h
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] [Equation] (cid:35) where s k +1 ∼ P ( · | s k , a k ) and a k = π k ( s k ) .
Furthermore, we write V ∗ and Q ∗ to denote optimal value functions obtained by maximizing over π (and are attained by the optimal policy π ∗ ).
When a learner plays π 1 , . .
, π N in M , we deﬁne its regret as
[Equation]
We use (cid:9) to denote a terminal state.
We let τ π denote the (random) trajectory generated by π .
For a state s and length- H trajectory τ , we write s ∈ τ π if s h
= s for some h ∈
[ H ] .
We deﬁne ( s, a ) ∈ τ
π similarly.
Finally, given an MDP M , M (2 H ) denotes a copy of M with a doubled horizon.
S ETTING
We work in the tabular meta-RL setting.
The learner can access T meta-training MDPs { ( S , A , P t , r t , H ) } t ∈ [ T ] =
{M t } t ∈ [ T ] .
Note that P t and r t both vary across tasks.
We set S := |S| and A := |A| .
After interacting with these tasks, the learner is presented with a meta-test MDP ( S , A , P Tg , r Tg , H ) , where the learner seeks to minimize its regret.
We assume, without loss of generality, that the MDPs have a shared starting state s 0 .
For meta-learning to succeed, the MDPs must have shared structure.
We focus on the following notion of shared hierarchical structure:
Deﬁnition 3.1 (Latent Hierarchy) .
Let { Z c } be a partition of the state space S into clusters.
We associate with each cluster Z a set of entrances Ent( Z ) ⊆
Z and a set of exits Ext( Z ) ⊆
Z × A .
We say that the tasks have a latent hierarchy with respect to ( { Z c } , Ent( · ) , Ext( · )) if for any Z c : (a
For any ( s, a ) ∈ ( Z c ×A ) \ Ext( Z c ) , P t ( · | s, a ) is constant over t and supported on Z c .
(cid:5)
(b) For any ( s, a ) ∈
Ext( Z c ) , there exists t, t (cid:48) with t (cid:54) =
t (cid:48) such that P t ( · | s, a ) (cid:54) =
P t (cid:48) ( · | s, a ) .
Furthermore, P t ( · | s, a ) is supported on (cid:83) c
Ent( Z c ) for any t ∈ [ T ] .
Deﬁnition 3.1 partitions the shared state space of the MDPs into clusters such that (1) non-exit ( s, a ) dynamics do not change between the MDPs and (2) exits are bottlenecks between clusters.
Example 3.1 (Gated Four-Room) .
Figure 1 illustrates the gated four-room environment along with a sample task.
The environment has a latent hierarchy with respect to the rooms outlined by colored gates (which can be open or closed). Entrances are colored aqua, while exits are marked by arrows.
Although we assume a single ﬁxed starting state, we can model task-dependent initial states by appending a dummy state s 0 .
A dummy action a 0 then takes the agent to the starting state for the task.
Observe that ( s 0 , a 0 ) is an exit, and therefore must transition to an entrance.
[Equation]
To see how Deﬁnition 3.1 captures intuitive notions of hierarchy in practical settings, we provide an example of a continuous setting roughly ﬁtting into our framework:
Example 3.2 (The Alchemy benchmark) .
Alchemy (Wang et al., 2021) is a recently proposed empirical benchmark for meta-RL, where the agent needs to place a stone in a series of potions to obtain some desired appearance, as illustrated in Figure 2.
Dipping a stone into a potion traverses an edge (determined by the potion) in a graph where nodes are possible stone appearances.
We focus on task distributions that randomize the edges of this graph (i.e., potion positions and feasible stone appearances are ﬁxed).
Then, the set of obtainable MDPs has a latent hierarchy where dipping the stone into any of the potions is an exit.
Indeed, other than dipping the stone into a potion, all other actions (e.g., moving the stone around the room) have the same dynamics in all tasks.
(cid:121)
For convenience, we will deﬁne several relevant notions.
First, for any cluster Z , we deﬁne its interior Z ◦ :=
( Z ×A ) \ Ext( Z ) .
Furthermore, we let Ent( S ) :
= (cid:83) c
Ent( Z c ) denote the set of all entrances and Ext( S ) :
= (cid:83) c
Ext( Z c ) the set of all exits.
Finally, we deﬁne
[Equation]
so that K and L are the total number of exits and entrances, respectively, while M is the maximal number of exits from any cluster.
[-= 
-]
Query Model.
We work in the online setting, where the agent interacts with the tasks by playing policies from the initial state s 0 .
During meta-training, we allow the agent to interact with the environments using an unbounded number of timesteps for each trajectory before resetting.
We then compute query complexity in terms of the total number of timesteps spent in all tasks in total.
M ETA -T RAINING A NALYSIS
In this section, we outline an algorithm for uncovering the latent structure that can be used for downstream tasks.
Recall that exits are deﬁned by changing dynamics between tasks.
To quantify the number of samples needed for exit detection, we deﬁne the following quantity: Image from Wang et al. ), extracted from a larger with no other
Deﬁnition 4.1 ( β -dynamics separation) .
There exists β > 0 such that for any t, t (
cid:48) ∈
[ T ] and ( s, a ) ∈
Ext( S ) , P t ( · | s, a ) (cid:54) = P t (cid:48) ( · | s, a ) =
⇒ (cid:107) P t ( · | s, a ) − P t (cid:48) ( · | s, a ) (cid:107) TV ≥ β .
(cid:5)
In deﬁning β above, we ensure high-probability exit detection with ˜ O
( S/β 2 ) samples.
Note that there is a brute-force approach to learning the underlying structure: one can learn P t ( · | s, a ) for all ( s, a ) and t ∈ [ T ] , and iterate over ( s, a ) to check for changing dynamics.
This can be done with query complexity ˜
O ( TS 2 A/β 2 ) .
However, under reasonable “coverage” assumptions outlined in the next section, this query cost can be lowered to ˜ O
( TKS/β 2 ) .
4. D EFINING A N OTION OF C OVERAGE
In supervised meta-learning, “diversity conditions” ensure that the meta-training tasks reveal the underlying latent structure (Tripuraneni et al., 2020; Du et al., 2020).
We provide analogous conditions ensuring that M 1 , . . .
, M T “cover” the latent hierarchy.
Since solving max π V π ( s 0 ) requires fewer samples than learning P , we expect such conditions to provide sample complexity gains.
Visitation Probabilities and α -Importance.
Minimally, exits should be visited by optimal policies of the meta-training tasks for coverage.
We thus deﬁne the following notion:
Deﬁnition 4.2.
Fix an MDP M =
( S , A , P , r, H ) , and let ( s, a ) ∈ S × A .
Construct a modiﬁed
MDP M \ ( s, a ) , where ( s, a ) brings the agent to a terminal state with no reward.
Then, we say that ( s, a ) is α -important for M if V M\ ( s,a ) ,
[Equation]
The α -importance condition quantiﬁes the value gap between policies that can use ( s, a ) and those that cannot.
For example, consider the task in Figure 3.
Any π with V π
( s 0 ) > 0 must visit the marked ( s, a ) pair with some probability.
Therefore, ( s, a ) is V ∗ 0 ( s 0 ) -important.
This suggests that high α -importance implies high visitation probability by nearoptimal policies.
The following result, proven in Section A, formalizes this connection:
Lemma 4.1.
If ( s, a ) is α -important for M , then for any policy π ,
V ∗ ( s 0 ) − V π ( s 0 ) < ε implies that P (( s, a ) ∈ τ π ) > ( α − ε ) /H .
A Preliminary Coverage Assumption?
Lemma 4.1 suggests a simple coverage condition: for any ( s, a ) ∈
Ext( S ) , assume that there exists t, t (
cid:48) ∈
[ T ] so that ( s, a ) is α -important for t and t (cid:48) , and P t ( · | s, a ) (cid:54) =
P t (cid:48) ( · | s, a ) .
However, as the following example shows, this condition excludes natural settings:
Example 4.1.
In Example 3.1, gates are either open or closed.
For closed gates, the associated ( s, a ) pairs are unimportant for any goal.
Thus, if ( s, a ) is α -important for some t, t (cid:48) ∈
[ T ] with α > 0 , then P t ( · | s, a ) =
P t (cid:48) ( · | s, a ) , and no change in dynamics can be detected using π ∗ t and π ∗ t (cid:48) .
Figure 4 illustrates how the proposed condition fails in covering an exit marked with a black arrow.
While ( s, a ) is unused by optimal policies in the ﬁrst task, it is V ∗ 0 ( s 0 ) -important for the second.
(cid:121)
Optimistic Imagination as a Coverage Mechanism.
The proposed assumption fails because there are cases where exits are only α -important in certain conﬁgurations (e.g., only open corridors are α -important in Example 3.1).
In such cases, near-optimal policies only ever see one conﬁguration of the dynamics for those exits and thus are insufﬁcient for formalizing exit coverage.
As an alternative, consider the following hypothetical scenario in the context of Figure 4: an agent has solved both tasks, achieving optimal values V ∗ 1 and V ∗ 2 .
Additionally, in the process of learning the second task, the agent has learned P 2 ( · | ⇓ ) .
If the agent then relearns the ﬁrst task while setting ˆ P 1 ( · | ⇓ ) ← P 2 ( · | ⇓ ) , it would obtain a new value ˆ V 1 (cid:29) V ∗ 1 .
Thus, it can reasonably conclude that the black arrow must have been an exit.
We illustrate this process in Figure 5, where π I is the optimal policy after “borrowing dynamics.”
The learner could then run π I for exit detection.
We refer to the counterfactual reasoning about the dynamics used above as optimistic imagination .
Unlike the preliminary condition, optimistic imagination only requires that an exit be important for one task and induce a value gap in another when borrowing dynamics – a weaker condition in many cases.
With the above intuition in mind, we now present the main coverage assumption.
Assumption 4.1 ( ( α, ζ ) -coverage) .
Assume ( M t ) t ∈
[ T ] have a latent hierarchy with respect to ( { Z k } , Ent( · ) , Ext( · )) .
There exists α, ζ > 0 such that for any { ( s 1 , a 1 ) , . . .
, ( s n , a n ) } ⊆
Ext( S ) , (a) For any i ∈ [ n ] , ( s i , a i ) is α -important for some meta- training MDP M i .
(b) For some M t with t ∈
[ T ] , if we construct a new MDP ¯ M t
=
( S , A , ¯ P , r t , H ) via ¯ P ( · | s, a ) = (cid:26) P M i ( · | s, a ) ( s, a ) =
( s i , a i )
P M t ( · | s, a ) otherwise , i.e. we replace ( s i , a i ) dynamics with those from M i , then V ¯ M t , ∗ ( s 0 ) >
V M t , ∗ ( s 0 ) + ζ .
Informally, ( α, ζ ) -coverage ensures that optimistic imagination can borrow dynamics for unknown exits from other tasks to ﬁnd a better optimal policy. 2 Most reasonable task distributions satisfy ( α, ζ ) -coverage with enough tasks – we provide a heuristic explanation in the appendix.
4. A LGORITHM O UTLINE
In this section, we outline the algorithm that we use to detect exits.
Our procedure can be naturally divided into three phases: a task-solving phase, a reward-free phase, and an exit detection phase.
Throughout, we illustrate our steps in Figure 5, showing how Phases I and II allow the learner to ﬁnd the imagined policy π I .
The full details of the algorithm are provided in Section A.1.
4.2. P HASE I: T ASK -S PECIFIC D YNAMICS L EARNING
First, we solve M 1 , . . .
, M T with UCBVI.
Using UCBVI regret bounds from Azar et al.
(2017) together with Lemma 4.1, we can guarantee that all α -important exits are sufﬁciently visited.
Thus, during optimistic imagination, the learner would be able to borrow high-quality estimates of exit dynamics from other tasks.
For example, a learner that has solved both tasks in Figure 4 can borrow open blue gate dynamics for use in the ﬁrst task during optimistic imagination, as in Figure 6.
4.2. P HASE II: R EWARD -F REE RL
In order to perform optimistic imagination, the learner also needs to simulate non-borrowed ( s, a ) dynamics.
This is done by fully learning the dynamics of one of the tasks, proving a template ˆ P 0 .
Learning ˆ P 0 is achieved using reward-free RL (Jin et al., 2020).
is insufﬁcient for learning π I , which requires the green region.
To understand the necessity of Phase II, note that in Figure 7, nearoptimal policies have no coverage over states past the blue gate (as shown by the red region).
Therefore, dynamics estimates from Phase
I are insufﬁcient for optimistic imagination.
On the other hand, by learning the dynamics fully in one of the tasks (and thus learning the green region), the learner can nevertheless simulate the dynamics if the blue gate were open and successfully recover π I .
4.2. P HASE III: E XIT D ETECTION
Having completed the previous two phases, the learner can use optimistic imagination to detect exits.
In particular, we consider a modiﬁed value iteration method where the learner optimistically chooses dynamics estimates from Phases I and II to perform Bellman backups.
This implicitly deﬁnes an MDP whose optimal value is at least as large as ¯ M t in Assumption 4.1, as ¯ M t would have been feasible for this process.
Analogously with α -importance, this value gap implies that the corresponding optimal policy π I for this new MDP must visit an ( s, a ) pair whose dynamics are borrowed.
Therefore, by playing π I , the learner can determine a new exit.
4. M ETA -T RAINING G UARANTEE
We now outline our main result for the algorithm in Section 4.2.
We ﬁrst deﬁne a “hierarchy oracle” that will be used in downstream tasks:
Deﬁnition 4.3 (Hierarchy oracle) .
Let (cid:9) S and (cid:9) F denote successful and failed termination, respectively.
Consider any tuple ( x, f, r, ˜ H ) such that x ∈ Ent( S ) , f : Ext( S ) → {(cid:9) S , (cid:9) F } , r is a reward function, and ˜ H ≤ H .
Such tuple induces an MDP M ( x, f, r, ˜ H ) =
[Equation] S ∪ {(cid:9) S , (cid:9) F } , A , P f , r, ˜ H ) whose starting state is x and whose transition dynamics is given by
[Equation]
P [Equation] a ) otherwise, for any t ∈ [ T ] .
An ε -suboptimal hierarchy oracle , when queried with any valid ( x, f, r, ˜ H ) , returns an ε -suboptimal policy for M ( x, f, r, ˜ H ) .
[Equation]
To motivate Deﬁnition 4.3, observe that learning a new MDP with the same latent hierarchy only requires visiting exits (as reward-free learning has coverage over cluster interiors). This, in turn, can be achieved by an agent that has a policy for reaching any exit from every entrance in any cluster.
We emphasize that the learner does not need to know the states in the actual clusters themselves . Formally, for an entrance x in cluster Z and exit e ∈ Ext( Z ) , we can query the hierarchy oracle with ( x, 1 [( · , · ) = e ] , 1 [ · = (cid:9) S ] , H ) to obtain the desired reaching policy. Disconnecting the clusters ensures that the learner cannot use any other exits, and thus the validity of the policy under any MDP that has the same latent hierarchy.
Our meta-training guarantee ensures that the hierarchy oracle is implementable:
Theorem 4.1 (Meta-training guarantee, informal) .
Under Assumption 4.1 and other assumptions in
Section A.2, the data obtained from the algorithm in Section 4.2 allows for: (a) implementing an ε -suboptimal hierarchy oracle, and (b) determining, for every s ∈ Ent( S ) , the reachable exits in the cluster containing s , simultaneously with probability at least 1 − p .
Furthermore, this is achieved with query complexity ˜
O
(cid:20) T (cid:18)
KL
[Equation]
KS
[Equation] SA [Equation] As a point of comparison, we have the following guarantee on brute-force hierarchy learning:
Theorem 4.2.
The brute-force approach outlined in Section A.5, under Assumption 4.1(a), determines the set of exits with high probability and query complexity
[Equation]
When α , β , and ζ are of the same order, we see that the proposed method incurs a smaller query complexity compared to a brute force learner that has only learned the exits.
We provide proofs of both results in Section A, along with all other necessary assumptions and full algorithm details.
M ETA -T EST A NALYSIS
In this section, we provide regret bounds on learning an MDP M Tg using the hierarchy oracle.
We ﬁrst characterize a family of tasks for which one can achieve improved regret bounds.
Furthermore, we provide sufﬁcient conditions ensuring that using the hierarchy incurs low suboptimality.
5. A SSUMPTIONS
In this section, we outline the assumptions that we make to prove a regret bound on the meta-test task.
Let M Tg =
( S , A , P Tg , r Tg , H ) be the meta-test MDP.
We assume that P Tg ( · | s, a ) =
P t ( · | s, a ) for any ( s, a ) (cid:54)∈ Ext( S ) and t ∈ [ T ] .
Our ﬁrst assumption restricts our attention to meta-test tasks which are compatible with the hierarchical structure:
Assumption 5.1 (Task Compatibility) .
There exists a cluster Z ∗ such that r
Tg is supported on ( Z ∗ ) ◦ ∪ Ext( S ) .
Furthermore, there exists an optimal policy π ∗ satisfying (a) Conditioned on s h ∈ Z ∗ , we have that ( s h (cid:48) , a h (cid:48) )
(cid:54)∈ Ext( Z ∗ ) for h (cid:48) ≥ h almost surely.
(b) The number of exits encountered by π
∗ is bounded by H eﬀ with probability ζ .
Hierarchical compatibility.
Intuitively, Assumption 5.1 suggests that the task can be decomposed into a ( Z ∗ ) -searching phase and a within- ( Z ∗ ) phase.
This decomposition captures the goalconditioned RL setting, which has been studied extensively in recent empirical works (Nachum et al., 2019a; Levy et al., 2018; Nachum et al., 2018).
We expect the hierarchy oracle to reduce the complexity of exploration in both phases.
Thus, these conditions ensure compatibility with the learned hierarchy.
Temporal Abstraction.
Since a hierarchical learner makes decisions upon entering a new cluster (to decide which exit to use/whether to stay), hierarchical planning can reduce the planning horizon.
Condition (b) quantiﬁes this reduction.
Note that in most practical settings, a learner that has access to good sub-skills can achieve low failure probability ζ even with modest values of H eﬀ .
Hierarchical Suboptimality.
Restricting the learner to executing hierarchical policies can greatly reduce the search space.
While this reduction leads to improved regret bounds, this also incurs approximation error, as the optimal policy may not lie in this restricted class.
We refer to this error as hierarchical suboptimality .
We will show that hierarchical suboptimality is controllable with appropriate conditions on P T , which require the following notions of reaching times:
Deﬁnition 5.1 (Reaching times) .
Fix a cluster Z , starting and goal states s, g ∈ Z , and planning horizon ˜
H ≤ H .
For any policy π , let ( s 0 , s 1 , . . .
, s ˜ H ) be the states visited by π from s , where s
=
s .
Then, we deﬁne
[Equation]
T min
[Equation] [Equation] (cid:5) In words, T π ˜ H ( s, g ) is the time π takes to reach a state g from s while remaining within the same cluster.
By minimizing this quantity in expectation over all policies, we obtain T ∗ ˜ H
( s, g ) .
Finally,
T min ( s, g ) is the minimum time for which it is possible to reach g from s .
Deﬁnition 5.2 (Regular and low-variance dynamics) .
There exists α, β, γ > 0 such that for any cluster Z , states s, g ∈ Z , and horizon ˜
H < H ,
(a) ( ( α, β ) -unreliability)
For any deterministic policy π with E [ T π ˜ H ( s, g )] − T ∗ ˜ H
( s, g ) <
α , T π ˜ H ( s, g ) has a sub-Gaussian upper tail with variance proxy β 2 E
[ T π ˜ H ( s, g )] 2 .
(b) ( γ -goal-reaching suboptimality) T ∗ ˜ H
( s, g ) ≤ (1 + γ ) T min ( s, g ) .
[Equation]
To understand ( α, β ) -unreliability, note that the condition only considers near-optimal deterministic policies.
Therefore, (a) quantiﬁes the randomness in T π
( s, g ) derived from the transition dynamics.
On the other hand, γ -goal-reaching suboptimality quantiﬁes the regularity of the dynamics, measuring whether near-optimal goal-reaching policies nearly achieve the optimal expected reaching time.
Deterministic environments satisﬁes these conditions with α = ∞ and β = γ = 0 .
We provide an extended discussion of why these quantities control hierarchical suboptimality in Section B.5.
Note that the guarantees of Deﬁnition 5.2 scales with the cluster width, and thus we have the following ﬁnal assumption:
Assumption 5.2.
For any cluster Z and s, g ∈ Z with s (cid:54) =
g , T min ( s, g ) ≤ W .
Furthermore,
H eﬀ W (cid:28) H .
Assumption 5.2 limits the length of the subtasks within each cluster.
This is consistent with hierarchy-based methods in practice, with skills only being executed for a limited amount of time.
This width bound, together with Assumption 5.1, suggests that π ∗ requires O ( H eﬀ W ) timesteps with high probability.
Therefore, the condition H eﬀ W (cid:28) H means that the task horizon is much longer than the minimum time required to complete the task, which often holds in practice.
5. M ETA - TEST R EGRET G UARANTEE
We note that Assumption 5.1 implies that the hierarchy oracle induces a “high-level” MDP. We have the following regret bound on applying E ULER to this MDP:
Theorem 5.1.
We work under Assumptions 5.1 and 5.2.
Furthermore, assume that the learner has access to an ε -suboptimal hierarchy oracle as guaranteed by Theorem 4.1, where ε < α .
Then, a learner that applies the procedure in Section B.2 to M Tg incurs regret
[Equation]
with high probability.
Observe that the irreducible hierarchical suboptimality ε subopt (i.e. when ε = 0 ) tends to zero as γ, β, ζ → 0 .
In particular, environments with deterministic in-cluster dynamics do not incur hierarchical suboptimality.
We prove this regret bound in Section B.
When does knowing the hierarchy help?
Consider the binary tree environment in Figure 8.
All of the leaves take the learner to a state with exits with probability 1 / 2 , with the exception of a special leaf (cid:96) ∗ that does so with probability (1 / 2)+ ε .
Rewards can only be collected upon performing one of the exits (blue/purple).
To achieve low regret, a learner has to quickly identity (cid:96) ∗ and the correct exit.
We consider the set of task distributions indexed by (cid:96) ∗ that randomize the reward-granting exit.
Knowing the hierarchy amounts to knowing (cid:96) ∗ , reducing the exploration problem to determining the correct exit.
However, a hierarchy-oblivious learner needs to explore the tree, leading to regret that is exponential in the tree depth.
Formally, we have the following result:
Theorem 5.2.
There exists a family of task distributions such that any hierarchy-oblivious learner incurs expected regret lower bounded by Ω(2 W/ 2 √ H 2 N ) on at least one task distribution.
In contrast, a learner with access to a 0 -suboptimal 4 hierarchy oracle incurs regret bounded by O ( √ H 2 N ) with high probability, over any sampled task from any of the task distributions.
We prove this result in Section B.4.3, using recent results by Domingues et al.
(2021) which demonstrate that the set of binary tree subproblems above form a set of minimax instances for any RL algorithm.
This separation result suggests that hierarchy-based learners gain in situations where temporally extended exploratory behaviors are needed.
This corroborates the experimental ﬁndings of Nachum et al.
(2019b), which attributes the beneﬁts of hierarchical RL to improved exploration.
R ELATED W ORK
Hierarchical reinforcement learning has been studied extensively (Sutton et al., 1999; Parr & Russell, 1998;
Dietterich et al., 1998; Vezhnevets et al., 2017). An early approach formalizing the use of hierarchies in RL is the options framework (Sutton et al., 1999), which ﬁxes a ﬁnite set of available skills/options. Since then, a large body of work has focused on designing methods for learning and adapting these options during the learning process (McGovern & Barto, 2001; Menache et al., 2002; ¸Sim¸sek & Barto, 2004; Mann et al., 2014). Of particular note is the work of Frans et al.
(2018), which learns a ﬁnite set of neural network sub-skills in the meta-RL setting.
On the other hand,
Laplacian-based option discovery in Machado et al.
(2017; 2018) deﬁnes options using proto-value functions (Mahadevan, 2005), which captures global features of the state space.
In more theoretical directions, Fruit & Lazaric (2017); Brunskill & Li (2014) provide regret and sample complexity bounds, respectively, for learning with options.
Additionally, Mann & Mannor (2014) demonstrate that options can improve the convergence rate of approximate value iteration.
More recent empirical work has considered the problem of hierarchy learning beyond the options framework in a wide variety of settings.
Nachum et al. (2019a); Levy et al.
(2018) provide algorithms for learning hierarchies based on goal-conditioned policies, reducing the learning problem to choosing subgoals.
Nachum et al.
(2018) considers a more general case when learned representations are used to map states to goals.
Other works such as Co-Reyes et al.
Eysenbach et al.
; Sharma et al.
(2019) provide intrinsic objectives for learning hierarchies without rewards.
Closely related is the work of Wen et al.
(2020), which introduces a similar latent structure on the state space, and provides sufﬁcient conditions ensuring sample-efﬁcient and tractable hierarchybased learning.
However, their works hinges on prior knowledge of the latent structure, while a major focus of our work is discovering the structure itself from interactions.
Several heuristics have been proposed in prior work for the detection of useful bottlenecks in MDPs (McGovern & Barto, 2001; Menache et al., 2002; ¸Sim¸sek & Barto, 2004; ¸Sim¸sek et al., 2005).
In particular, ¸Sim¸sek & Barto (2004) deﬁne the notion of access states , states which maximize shortterm novelty of future states.
Although exits as deﬁned in Deﬁnition 3.1 meet this heuristic, we note that cluster interiors may also contain bottlenecks also meet these conditions.
In contrast, our algorithm would not detect these bottlenecks; however, this behavior is desirable as such bottlenecks are unimportant for meta-learning.
C ONCLUSION
We have demonstrated that certain natural coverage conditions allow for learning useful hierarchies from tasks.
Interesting future directions include analyzing hierarchy-based multi-task RL and extending the ideas in this work to continuous state and/or action spaces.
Another interesting direction would be to provide sample-efﬁcient algorithms for learning additional structures that can be imposed on the learned hierarchy, such as cluster equivalences as in Wen et al.
A M ETA -T RAINING P ROOFS A. A LGORITHM
In this section, we provide the complete algorithm for exit detection with optimistic imagination.
For readability, we separate the three phases.
A.1. P HASE I: T ASK -S PECIFIC L EARNING
Require: Tasks M 1 , . . .
, M T , N UCBVI UCBVI iterations, N TS policy samples, threshold N TSthresh 1: for all t ∈
[ T ] do 2:
D t ← ∅ .
Obtain policy set Φ t ← UCBVI( M t , N UCBVI ).
4: for all n = 1 , . . .
, N TS do 5:
Sample π ∼ Unif (Φ t ) .
Play π in M t , add all ( s, a, s (cid:48) ) pairs to D t , get sum of rewards ˆ V ( n ) .
Form estimated dynamics model ˆ P t from D t .
Form optimal value estimate ˆ V t ←
N TS
N TS (cid:88) n
= ˆ V ( n ) 9:
N t ( s, a ) ← |{ ( x, u, x (cid:48) )
∈
D t | x = s, u = a }| .
10: for all ( s, a ) ∈ S
× A do 11: if N t ( s, a ) < N TSthresh then ˆ P t ( · | s, a ) ← 0 .
12: return dynamics estimates ˆ P t and value estimates ˆ V t for t ∈ [ T ] .
A.1. P HASE II: L EARNING R EFERENCE D YNAMICS
Require:
MDP M , N RF E ULER Euler iterations, N RF policy samples 1: Set policy class Ψ ← ∅ and dataset D RF ← ∅ 2: for all g ∈ S do 3:
Create MDP M g from M with horizon 2 H and P ( (cid:9) | g, a )
A.1.
P HASE III: E XIT D ETECTION
Require: N ED , N EDthresh , N EL E ULER , N EL policy samples 1: Initialize I S E XIT [ s, a ] ← F ALSE for ( s, a ) ∈ S × A .
2: while True do 3: for all t ∈ [ T ] do 4: ˆ P 0 ( · | s, a ) ← ˆ P t ( · | s, a ) for ( s, a ) ∈ S
× A with I S E XIT [ s, a ] .
5: ˜ V t , ˜ Q t ← O PT I MG VI( ˆ P 0 , (ˆ P 1 , . . .
, ˆ P T ) , r t , I S E XIT ) 6: if ˜ V t 0 ( s 0 ) − ˆ V t > (2 / 3)
ζ then 7:
Run greedy policy with respect to ˜ Q N ED times and form estimate ˆ P for ( s, a ) pairs visited at least N (cid:48) thresh times.
8: for all ( s, a ) ∈ S
×
A with ˆ P ( · | s, a ) (cid:54)≡ 0 do 9: if ( ∃ t ∈ [ T ]) ˆ P t ( · | s, a ) (cid:54)≡ 0 and (cid:13)(cid:13)(cid:13) ˆ P ( · | s, a ) − ˆ P t ( · | s, a ) (cid:13)(cid:13)(cid:13) TV > β/ 2 then 10:
I S E XIT [ s, a ] ← True 11: ˆ P t ( · | s, a ) ← L EARN -E XIT ( M t , ( s, a ) , N EL E ULER , N EL ) for all t ∈
[ T ] 12: if no new exits found after passing through T tasks since last found exit then 13: return I S E XIT
: procedure BOAT-VI(Reference dynamics ˆ P 0 , Estimated dynamics (ˆ P 1 , . .
, ˆ P T ) ,
Reward function r , Table I S E XIT
[ S × A ]) 2: ˆ V H ( s ) ← 0 for s ∈ S .
3: for all h
= H − 1 , . . .
, 0 do 4: for all ( s, a ) ∈ S
× A do 5: if I S E XIT [ s, a ] then 6: ˆ Q h ( s, a ) ← r ( s, a ) + ˆ P
ˆ V h
+
( s, a ) 7: else 8: ˆ Q h ( s, a ) ← r ( s, a ) + max t =0 ,...,T ˆ P t
ˆ V h
+1 ( s, a ) 9: ˆ V h ( s ) ← max a ˆ Q h ( s, a ) for s ∈ S .
10: return ˆ V , ˆ Q
1: procedure L EARN -E XIT (MDP M , exit ( s, a ) , N EL E ULER E ULER iterations, N EL policy samples) 2:
Create MDP ˜ M from M so P ( (cid:9) | s, a ) = 1 .
3: ˜ r ( s (cid:48) , a (cid:48) ) ← 1 [( s (cid:48) , a (cid:48) )
=
( s, a )] for any ( s (cid:48) , a (cid:48) )
∈ ( S ∪ {(cid:9)} ) × A .
4: Ψ ← E ULER ( ˜ M , ˜ r, N EL E ULER ) 5: for all n = 1 , . . .
, N EL do 6:
Sample π ∼ Unif (Ψ) .
Play π in M and obtain trajectory ( s 0 , a 0 , . . .
, s H ) .
8: return reference dynamics ˆ P ( · | s, a ) formed from all trajectory data.
A. O THER A SSUMPTIONS AND R ELEVANT D EFINITIONS
The remaining assumptions quantify the reachability of certain states.
First, we have the following assumption, which in effect ensures that one can reach most states regardless of exit conﬁguration in the meta-training tasks:
Assumption A.1 (Non-limiting exit conﬁgurations) .
Let M =
( S , A , P , H ) be any reward-free environment with time-varying dynamics
[Equation]
Then, there exists C > 1 such that for any s and t ∈
[ T ] ,
[Equation]
Intuitively, the assumption states that the reachability of a state in M t would not be signiﬁcantly improved even under an optimal conﬁguration of the exits.
Therefore, running reward-free RL on one of the meta-training tasks is sufﬁcient for learning all non-exit ( s, a ) pairs.
Remark A.1.
We note that Assumption A.1 is restrictive in that it requires that every state be roughly reachable in any of the meta-training MDPs.
This may not hold in practice, e.g., consider a four-room environment where one of the rooms is blocked off for one of the tasks.
However, this can be weakened to requiring that s be reachable in at least one of N arbitrarily chosen meta-training tasks.
This would require that the algorithm run Phase II over N meta-training tasks, which results in a benign increase in the query complexity of the algorithm, so long as N is a constant much smaller than T .
We focus on the N = 1 case for ease of presentation.
[Equation]
To simplify the presentation of the rest of the assumptions, we recall the following deﬁnition of δ -signiﬁcance in Jin et al.
Deﬁnition A.1.
A state s is δ -signiﬁcant if max π P ( s ∈ τ π ) ≥ δ .
Additionally, we say that ( s, a ) is δ -signiﬁcant if s is δ -signiﬁcant.
(cid:5)
Note that we have modiﬁed the deﬁnition to remove the dependence on the timestep h ∈
[ H ] .
This is because the dynamics are stationary, and thus it does not matter when s is visited in a trajectory.
Note that Assumption A.1 implies that a reachable entrance in one task is reachable in all tasks (i.e., is signiﬁcant in the sense of Deﬁnition A.1). Therefore, we can quantify the minimum level of signiﬁcance:
Deﬁnition A.2 ( ρ -signiﬁcant entrances) .
For any s ∈ Ent( S ) , s is ρ -signiﬁcant for any task.
(cid:5)
Finally, we want to quantify the reachability of every exit from any entrance in the same cluster, assuming that it is indeed reachable:
Deﬁnition A.3 (In-cluster exit reachability) .
Fix any cluster Z , entry s ∈ Ent( Z ) , and exit ( g, a
) ∈
Ext( Z ) .
Consider the reward-free environment M t | Z
=
( Z, A, P t | Z , H ) , where P t | Z is the restriction of P t to Z × A and the starting state is s .
Then, if g has nonzero signiﬁcance in M t | Z for any t ∈ [ T ] , then it is ε 0 -signiﬁcant.
(cid:5)
The requirement that the assumption hold for any t ∈
[ T ] is without loss of generality since non-exit dynamics do not change.
A. V ERIFYING E XIT D ETECTION
In this section, we demonstrate that the algorithm in Section A.1 can successfully discover Ext( S ) with high probability.
Formally, we have the following result:
Theorem A.1 (Provable exit detection) .
Assume we run the algorithm in Section A.1 with the parameter choices given in Table 1.
Then, with probability at least 1 − p , the algorithm returns an array I S E XIT satisfying:
[Equation]
To prove this result, we proceed with a phase-by-phase analysis of the algorithm in Section A.1, which we then compile into proof of the desired result.
[Table]
A.3. P HASE I A NALYSIS
First, we prove that during Phase I, Algorithm 1 sufﬁciently visits all relevant exits and that all value estimates are sufﬁciently close.
Formally, we have the following result:
Proposition A.1.
Set
[Equation]
and consider the following procedure applied to one of the meta-training tasks M t :
UCBVI is run for
[Equation]
iterations, generating policies
π
[Equation]
, π ( t ) N UCBVI
[Equation]
The learner uniformly samples
[Equation]
policies from the previous step, runs each policy in M t , and obtains a dataset of transitions D t and returns ˆ V (1) , . .
, ˆ V ( N TS ) .
Then, with probability at least 1 − p/ 3 T ,
(a) We have the regret bound
[Equation]
(b) The set of obtained returns satisfy
[Equation]
(c) If ( s, a ) is α -important for M t , then N t ( s, a ) ≥ N thresh .
(d) For every ( s, a ) pair such that N t ( s, a ) ≥ N thresh ,
[Equation]
To prove the above result, we ﬁrst recall the following regret bound on UCBVI, as proven by Azar et al.
Lemma A.1 (UCBVI regret bound) .
For sufﬁciently large N , with probability at least 1 − p/ 6 , V [Equation] ( s
[Equation]
As we will see later on, with our choice of N UCBVI , we obtain the desired regret bound in (a).
Additionally, by Hoeffding’s inequality, the average of N TS returns concentrates around the desired quantity with high probability, proving (b).
Thus, all that remains is ensuring that every α -important exit is sufﬁciently visited, and thus their dynamics are sufﬁciently well-estimated.
Recall from the main text that the key step is demonstrating that a near-optimal policy for a task must visit its α -important states with non-negligible probability:
Lemma A.2.
Let ( s, a ) be α -important for M , and let π be an ε -suboptimal policy for ε < α .
Then,
[Equation]
Proof.
By α -importance,
[Equation]
Therefore, by applying Lemma A.17 and noting that { ∆ ∩ τ π (cid:54) = ∅ } = { ( s, a ) ∈ τ π } , we obtain the desired result.
Through the prior result, we can relate the UCBVI regret bound to the probability that a randomly chosen UCBVI-generated policy visits an α -important state:
Lemma A.3.
Let ( s, a ) be α -important for M .
Assume that UCBVI , when run for
[Equation]
iterations, generates policies π 1 , . .
. , π N .
If we sample π uniformly from these policies and let τ be the (random) trajectory generated by this randomly selected policy, then
[Equation]
conditioned on the high probability event in Lemma A.1.
Proof.
By Lemma A.2, for any ﬁxed π , we can write
[Equation]
where x + = x 1 [ x > 0] .
Then, since π is chosen randomly from the policies generated by UCBVI,
[Equation]
Therefore, by applying the regret bound in Lemma A.1 and the choice of N , we ﬁnd that
[Equation]
With all of the above intermediate results, we can now prove Proposition A.1.
Proof of Proposition A.1.
Throughout this proof, we condition on the high-probability event in
Lemma A.1, instantiated to occur with probability at least p/ 12 T .
(a) By the choice of N UCBVI ,
[Equation]
and thus, we obtain the desired bound by plugging this value into the regret bound provided by Lemma A.1.
(b) Note that ( ˆ V ( i ) ) are i.i
d., bounded in [0 , H ] , and for any i ∈
[ N TS ] ,
[Equation]
Therefore, by applying Hoeffding’s inequality, with probability at least 1 − p/ 12 T ,
[Equation]
[Equation]
The result immediately follows from the fact that N TS (cid:38) ( H 2 /ζ 2 ) log( T/p ) .
(c)
The result simply follows from Lemma A.16 instantiated with failure probability 1 − p/ 12 T , together with the choice of N TSthresh .
(d) With the choice of N UCBVI , the conclusion of Lemma A.1 can be made to hold with probability at least 1 − p/ 24 T .
Fix an α -important exit ( s, a ) for M t , so that the probability that ( s, a ) is visited by the procedure is at least α/ 2 H .
By Lemma A.15, sampling N TS trajectories is sufﬁcient to ensure that N t ( s, a ) ≥ N TS thresh with probability at least 1 − p/ 24 TK .
Therefore, by performing a union bound over the set of α -important exits (which contains at most K elements), N t ( s, a ) ≥ N TSthresh for any α -important exit with probability at least 1 − p/ 24 T .
Thus, overall, this event occurs with probability at least 1 − p/ 12 T .
Since each part fails with probability at most p/ 12 T , the overall failure probability is at most p/ 3 T , the desired result.
A.3. P HASE II A NALYSIS
In this section [Equation] we provide guarantees on the dataset D RF obtained by performing reward-free RL in Algorithm 2.
Formally, we have the following high-probability result:
Proposition A.2.
For any δ > 0 and failure probability p , if Algorithm 2 is run with parameters
[Equation]
Then, with probability at least 1 − p/ 3 :
(a) The distribution µ generating each sample in D RF satisﬁes
[Equation]
(b) The estimated dynamics model ˆ P 0 satisﬁes
[Equation]
The details of the proof of Proposition A.2 follow that of Jin et al.
(2020), which we provide here for completeness.
First, we adapt the regret bound from Zanette & Brunskill (2019) for any MDP and reward function used in Algorithm 2.
Lemma A.4.
For any g ∈ S , running E ULER in M g for N iterations returns N policies π 1 , . . .
, π N satisfying the regret bound
[Equation]
with probability at least 1 − p .
Proof.
Observe that
[Equation]
Therefore, by applying the regret bounds from Zanette & Brunskill (2019)
[Equation]
we obtain the regret bound
[Equation]
with probability at least 1 − p .
With the regret bound above, we now proceed to prove Proposition A.2.
Proof of Proposition A.2.
(a) Fix a δ -signiﬁcant g ∈ S .
Note that for r g , V π 0 ( s 0 ) =
P ( g ∈ τ π ) for any policy π .
Therefore, via the regret bound from Lemma A.4 and the choice of N RF E ULER , we obtain
[Equation]
with probability at least 1 − p/ 2 S .
Now, since π ( · | g ) ∼ Unif ( A ) , we have that for any a ,
[Equation]
Finally
[Equation]
by applying the same argument above across all δ -signiﬁcant g ∈ S
[Equation]
we have that for any ( g, a ) ,
[Equation]
with probability at least 1 − p/ 2 .
To complete the proof of (a), observe that
[Equation]
since conditioned on ( g, a ) ∈ τ π , the probability that ( g, a ) is sampled is at least 1 / 2 H .
(b) The result follows by following the same proof of Lemma C.2 in Jin et al.
(2020), with failure probability p/ 2 .
Note that the dynamics are stationary, and thus we do not need to perform a union bound over the time step
h ∈
[ H ] .
A.3. P HASE III A NALYSIS
Having analyzed the previous two phases, we now show that Algorithm 3 successfully ﬁnds all exits during Phase III.
As part of this, we prove the following guarantee:
Proposition A.3.
Assume that Algorithm 3 is at Line 3, having just arrived at this step for the ﬁrst time, or after ﬁnding a new exit.
Let E = { ( s, a ) | I S E XIT
[ s, a ] } .
We assume:
(a) E ⊆
Ext( S ) .
(b) The high-probability events in Proposition A.1 (for any t ∈
[ T ] ) and Proposition A.2 (for δ ≤ ζ/ 24 CH 2 S ) both hold, providing estimators ˆ P 0 , ˆ P 1 , . .
, ˆ P T .
(c)
For every ( s, a ) ∈ E and t ∈ [ T ] , we have access to an estimator ˆ P t ( · | s, a ) for P t ( · | s, a ) satisfying
[Equation]
Then, if E = Ext( S ) , the algorithm terminates after passing through T tasks.
Otherwise, if E (cid:54) =
Ext( S ) , the following events hold simultaneously with probability at least 1 − p/ 3 K :
(a) For one of the next T tasks that the algorithm inspects, there exists at least one t ∈ [ T ] such that
[Equation]
(cid:12)(cid:12)(cid:12)
> 2 3 ζ. (b) For the task in (a), running Lines 7–11 ﬁnds at least one ( s, a ) ∈
Ext( S ) \ E (and only ( s, a ) pairs in this set), and learns an estimator ˆ P t ( · | s, a ) for P t ( · | s, a ) satisfying
[Equation]
To prove the above result, we will consider the following special set of MDPs:
Deﬁnition A.4 (Imaginable MDPs) .
Fix a task M t .
Furthermore, let E ⊆
Ext( S ) .
For any function
I : S ×
A ×
[ H ] → { 0 , . .
, T } , we can construct an associated MDP M I =
( S , A , P I , r t , H ) via
[Equation]
We deﬁne the set of imaginable MDPs to be the set M t ( E ) to be the set of MDPs generated by any
I satisfying
[Equation]
Informally, M t ( E ) is the set of obtainable MDPs by borrowing dynamics for ( s, a ) pairs that are not known to be exits.
This set is of particular interest in our analysis, since BOAT-VI performs a maximization over the MDPs in this set:
Lemma A.5 (Optimism) .
Assume the preconditions of Proposition A.3.
Over the course of running
Algorithm 4, the algorithm implicitly deﬁnes an index function I : S ×A× [ H ] → { 0 , . . .
, T } .
This function I satisﬁes M I ∈
M t ( E ) , and M I is a maximizer of
[Equation]
Proof.
To see that
M I ∈
M t ( E ) , note that if ( s, a ) ∈ E , then I h ( s, a ) = t for any h ∈
[ H ] .
Otherwise, note that although the maximum is over all indices, ˆ P k ( s (cid:48) | s, a ) = 0 for any s (cid:48)
if
N t ( s, a ) < N thresh .
Therefore, since the estimated value function is always positive, the maximum is effectively only over any k with N k
( s, a ) ≥ 0 .
Thus, M I ∈ M t ( E ) .
Now, we prove that M I
= M is a maximizer of the estimated value function, which we prove by induction.
Let I (cid:48) be another index function satisfying M (cid:48)
=
M I (cid:48)
∈
M t ( E ) .
Clearly, ˆ V M , ∗ H ( s ) = 0 ≤ ˆ V M (cid:48) , ∗ H ( s ) .
Then, for any h ∈ [ H ] and ( s, a ) ,
[Equation]
where the ﬁrst inequality follows from the deﬁnition of I , and the second follows from the inductive hypothesis.
Therefore, for any s ,
[Equation]
Thus, by induction, ˆ V M , ∗
[Equation]
s 0 ) .
Since the argument applies for any I (cid:48) , we have shown the desired optimality result.
Note that ¯ M is contained in M t ( E ) via our assumed preconditions, suggesting that the BOATVI should ﬁnd an MDP with a sufﬁciently over-optimistic value.
However, the maximization above makes use of estimated dynamics, and thus we need to prove that every MDP in M t ( E ) is sufﬁciently well-estimated.
We now show that the preconditions of Proposition A.3 are sufﬁcient for estimation.
To this end, we recall the performance difference lemma:
Lemma A.6 (Performance Difference) .
Fix two MDPs M =
( S , A , r, P , H ) and M (cid:48) =
( S , A , r, P (cid:48) , H ) .
Then, for any policy π ,
V M (cid:48) ,
[Equation]
We now present the estimation result:
Lemma A.7.
For any π and t ∈ [ T ] , let V M ,π( s 0 ) be the value of a policy π in M ∈ M t ( E ) , and ˆ V M ,π
[Equation]
s 0 ) an estimate using available quantities from the preconditions of Proposition A.3.
Then,
[Equation]
Proof.
Fix a t ∈
[ T ] , M ∈ M t ( E ) and policy π , with associated index function I .
Lemma A. implies that
[Equation]
We now deﬁne the following sets:
[Equation]
insigniﬁcant for
[Equation]
Note that R h is estimated via reference dynamics from Phase II, while B h is estimated using taskspeciﬁc dynamics from Phase I.
Then, for a ﬁxed h , we can decompose the inner sum above as
[Equation]
Note the inequality since E ∩ U δ is not necessarily disjoint.
We now bound the four terms above separately.
Bounding (I): Dynamics Error from Known Exits.
We ﬁrst bound (I), which we note derives from errors in estimating the dynamics of known exits.
Recall that by precondition (c) in Proposition A.3, [Equation]
[Equation]
Therefore,
[Equation]
I)
[Equation]
) ≤ ζ 24 H .
Bounding (II): Reference Dynamics Error.
Note that within R h , P h = P 0 , which we estimate via ˆ P 0 .
Therefore, we bound the error resulting from using D RF to estimate P 0 .
This part of the proof follows that of Jin et al.
First, by Cauchy-Schwarz,
[Equation]
=
[Equation]
Observe that ˆ V πh +1 only depends on π through timesteps h
+ 1 , . .
, H − 1 .
Therefore,
[Equation]
By applying Assumption A
[Equation] [Equation]
where we have applied Assumption 4.1 to move from M to M 1 .
Substituting into the earlier expression,
[Equation]
Thus, by applying the bound on the right-hand side provided by Proposition A.2, (II) = [Equation] [Equation] h
[Equation]
Bounding (III): Error from Task-Speciﬁc Dynamics.
Recall that on B h , P h
= P k for some k (cid:54) = 0 .
Thus, (III) is the error resulting from dynamics estimation in Algorithm 1.
By following the same argument as that used to bound (I) and applying Proposition A.1, we ﬁnd that
[Equation]
Bounding (IV): Error from δ -Insigniﬁcance.
The remaining set of ( s, a ) pairs are those such that s is δ -insigniﬁcant in M 1 .
Note that
[Equation]
As a result,
[Equation]
By setting δ = ζ/ 24 CH 2 S when performing reward-free RL in Phase II, we thus ﬁnd that
[Equation]
Concluding .
By combining the bounds on (I) through (IV) and summing across h = 0 , . . .
, H − 1 , we ﬁnd that
[Equation]
Note that this argument simultaneously applies to any such M ; therefore, the desired conclusion follows.
The prior estimation result, together with Assumption 4.1, suggests that BOAT-VI should ﬁnd an
MDP that sufﬁciently overestimates the value of the task so long as not all exits have been found.
This ensures that the exit-ﬁnding routine is triggered.
Formally,
Lemma A.8.
Assume the preconditions of Proposition A.3, and that E (cid:54) =
Ext( S ) .
Additionally, let t ∈
[ T ] be the task with a ζ -overoptimistic value when borrowing exits Ext( S ) \ E .
Finally, let ˜ V t be the value function returned by Algorithm 4 on M t .
Then,
[Equation]
Proof.
Throughout this proof, we omit the timestep
and the initial state s 0 for brevity.
Deﬁne M ∗ and π ∗ to be the maximizers of
[Equation]
Furthermore, let ¯ M be the imagined MDP guaranteed by Assumption 4.1 on top of M t , such that
V ¯ M , ∗ > V M t , ∗ + ζ .
Then, we have that
[Equation]
Furthermore,
[Equation]
where the ﬁrst follows from optimality, while the second follows from Proposition A.2.
Thus, putting the two inequalities together,
[Equation]
While the prior algorithm ensures that at least one of the tasks will trigger the exit condition, the actual task that triggers the condition may not be the same one invoked in the proof above.
Nevertheless, we can prove that the trigger condition ensures that the algorithm will ﬁnd a new exit.
Lemma A.9.
Assume the preconditions of Proposition A.3, and that E (cid:54) =
Ext( S ) .
Let t ∈
[ T ] be a task such that the value estimate ˜ V t returned by Algorithm 4 satisﬁes ˜ V t 0 ( s 0 ) − ˆ V t >
(2 / 3) ζ , and let π be the optimal policy for ˜ V .
Then, there exists ( s, a
) ∈
Ext( S ) \ E such that for some t (cid:48) (cid:54) =
t , (a) N t (cid:48)
( s, a ) ≥ N TSthresh and P t ( · | s, a ) (cid:54) =
P t (cid:48) ( · | s, a ) .
(b) P M t (( s, a ) ∈ τ π ) >
ζ/ 6 KH .
Proof.
Let M be the implicit MDP deﬁned by Algorithm 4 in the process of computing ˜ V t .
We will prove a value gap between M and M t , which implies that π must visit state-action pairs with imagined dynamics.
Note that
[Equation]
where the ﬁrst inequality comes from Lemma A.7 and the last two inequalities come from Proposition A.1.
Thus, V M ,π( s 0 ) − V M t , ∗( s 0 ) ≥ ζ/ 6 .
We now leverage this value gap to show that π must use some exit ( s, a ) whose dynamics in M have been modiﬁed from M t with some probability.
Formally, deﬁne the set ∆ = (cid:8) ( s, a, h ) (cid:12)(cid:12) P t ( · | s, a ) (cid:54) =
P M h ( · | s, a ) (cid:9) .
By construction, ∆ ⊆
Ext( S ) ×
[ H ] , and for any ( s, a, h ) ∈ ∆ , there exists t (cid:48) such that N t (cid:48)
( s, a ) ≥ N TSthresh and P t (cid:48) ( · | s, a ) =
P M h ( · | s, a ) .
Furthermore, ∆ must be non-empty, as otherwise, P t = P Mh for all h , and thus V M ,π( s [Equation]
≤
[Equation]
( s [Equation] ) , a contradiction.
Therefore, by applying Lemma A.17, we ﬁnd that
[Equation]
which implies the desired result, as { ( s, a ) | ( s, a, h ) ∈ ∆ } has at most K elements.
Because of Lemma A.9, we simply need to run π enough times and threshold at the number of samples needed to reliably determine which ( s, a ) pairs have an O ( β ) change in TV distance between tasks.
Lemma A.10.
We work in the setting of Lemma A.9.
Set
[Equation]
Then, if we execute π within M t N ED and let N ( s, a ) be the number of times that ( s, a ) is played in this process, then with probability at least 1 − p/ 6 K , the following hold:
(a) For the ( s, a ) pair and task t (cid:48) in Lemma A.9, N ( s, a ) ≥ N EDthresh and (cid:13)(cid:13)(cid:13) ˆ P ( · | s, a ) − ˆ P t (cid:48) ( · | s, a ) (cid:13)(cid:13)(cid:13) TV > β 2 .
(b) For any ( s, a ) (cid:54)∈ Ext( S ) with N ( s, a ) ≥ N EDthresh and t (cid:48) with N t (cid:48) ( s, a ) ≥ 0 , (cid:13)(cid:13)(cid:13) ˆ P t ( · | s, a ) − ˆ P t (cid:48) ( · | s, a ) (cid:13)(cid:13)(cid:13) TV ≤ β 2 .
Proof.
By the choice of N ED and the lower bound P M t (( s, a ) ∈ τ π ) >
ζ/ 6 KH from Lemma
A.9, we guarantee that N ( s, a ) ≥ N EDthresh with probability at least 1 − p/ 12 K .
Furthermore, due to the choice of N EDthresh , with probability at least 1 − p/ 12 K , we have that for any ( s, a ) with
N ( s, a ) ≥ N EDthresh ,
[Equation]
by applying Lemma A.16.
We condition on these two events simultaneously for the rest of the proof, which occurs with probability at least 1 − p/ 6 K .
We now prove each part separately.
For brevity, we omit ( s, a ) wherever it is understood.
(a) By applying the triangle inequality,
[Equation]
Therefore, by lower bounding the left-hand side using β -dynamics separation in Deﬁnition 4.1, we ﬁnd that
[Equation]
(b) The triangle inequality implies that
[Equation]
where the bound on the ﬁrst term is provided by Proposition A.1.
The prior result demonstrates that if the exit-ﬁnding condition is detected at any point by the algorithm, then the algorithm ﬁnds a previously undiscovered exit in Ext( S ) .
At this point, all that remains is to ensure that the algorithm sufﬁciently learns the dynamics of the newly-found exit in all of the meta-training tasks.
Lemma A.11.
Fix an ( s, a
) ∈
Ext( S ) , which was found via exit detection, and let
[Equation]
Assume we run the exit-learning subroutine with
N EL E ULER = Ω
[Equation]
and N EL = Ω (cid:18) CH
[Equation]
in each of the tasks.
Then, with probability at least 1 − p/ 6 K ,
[Equation]
for every t ∈ [ T ] .
Proof.
Fix a task t ∈
[ T ] .
Note that Assumption 4.1 implies that ( s, a ) is ( α/H ) -signiﬁcant for some task.
Then, ( s, a ) must be ( α/CH ) -signiﬁcant
for all of the other tasks by Assumption A.1.
By applying Lemma A.4, the set of policies found by the exit-learning subroutine for every task t ∈ [ T ] satisﬁes
[Equation]
with probability at least 1 − p/ 18 TK .
By setting
[Equation]
we thus ﬁnd that for any t ∈ [ T ] ,
[Equation]
Note that the right-hand side is exactly the probability that the trajectory of a randomly chosen policy in Φ t ( s, a ) contains ( s, a ) in the trajectory.
Therefore, by applying Lemma A.15, playing
[Equation]
is sufﬁcient to guarantee that we obtain at least N ELthresh samples from P t ( · | s, a ) with probability at least 1 − p/ 18 TK .
Since ( s, a ) ∈ Ext( S ) , P t ( · | s, a ) (and by extension, ˆ P t ( · | s, a ) ) is supported on Ent( S ) .
Therefore, we can modify the proof in Lemma A.16 so that with probability at least 1 − p/ 18 TK we get the bound
[Equation]
with N ELthresh depending linearly on L instead of S .
Note that by performing a union-bound, all events occur with probability at least 1 − p/ 6 TK .
Performing a second union-bound over all of the available tasks results in the desired failure probability.
At this point, we have effectively proven the second half of our Phase III guarantee.
All that remains is to prove that if { ( s, a ) | I S E XIT [ s, a ] } =
Ext( S ) , then the algorithm terminates without triggering the exit-ﬁnding condition.
Lemma A.12.
Assume that E = { ( s, a ) | I S E XIT [ s, a ] } =
Ext( S ) .
Then, under the preconditions of Proposition A.3, every task satisﬁes
[Equation]
Proof.
Fix a task t ∈
[ T ] .
Once E = Ext( S ) , then M t ( E ) =
{M t } , since the only ( s, a ) -dynamics that can be substituted from other tasks are those of non-exits, which do not change between tasks.
Therefore, by Lemma A.5, ˜ V t ( s 0 ) =
ˆ V M , ∗( s 0 ) .
Finally, by applying Lemma A.7, we thus ﬁnd that
[Equation]
The desired result follows since the argument holds for any task t .
A.3. P ROOF OF T HEOREM A.
In this section, we compile the guarantees provided by each of the three phases into a proof of
Theorem A.1.
Proof of Theorem A.1.
We condition on the following high-probability events:
(a) Proposition A.1 guarantees for all M t with t ∈
[ T ] .
(b) Proposition A.2.
Via a union-bound, this holds with probability at least 1 − (2 / 3) p .
To prove the theorem, we provide an induction-based analysis of Phase III.
In particular, we will show that while E = { ( s, a ) | I S E XIT [ s, a ] } (cid:40) Ext( S ) , Phase III will add at least one state-action pair to E that belongs to Ext( S ) \ E .
Formally, let F k denote the internal state of the algorithm after it has added k state-action pairs.
Note that with k = 0 , { ( s, a ) | I S E XIT [ s, a ] } in F k is empty.
Thus, F k satisﬁes the preconditions of
Proposition A.3, which in turn implies that the algorithm adds a new state-action pair in Ext( S ) and sufﬁciently learns its dynamics for all tasks with probability at least 1 − p/ 3 K .
In short, the internal state of the algorithm at time F 1 also satisﬁes the preconditions of Proposition A.3 with probability at least 1 − p/ 3 K .
More generally, Proposition A.3 ensures that if F k satisﬁes the preconditions of
Proposition A.3, then so does F k +1 .
Therefore, with probability at least 1 − p/ 3 , F K satisﬁes the preconditions of Proposition A.3, which necessarily implies that { ( s, a ) | I S E XIT [ s, a ] } =
Ext( S ) in F K , and thus the algorithm exits as desired.
By performing a union bound, all this occurs with probability at least 1 − p .
[Equation]
Having demonstrated that Ext( S ) can be successfully recovered by interacting with the environment, we now show that the data can also be used to determine exit reachability and implement the hierarchy oracle.
We formally state our main result here:
Theorem A.2.
Assume that M 1 , . .
, M T have a latent hierarchy with respect to ( { Z c } , Ent( · ) , Ext( · )) , and assume that these tasks satisfy the ( α, ζ ) -coverage condition in
Assumption 4.1.
Furthermore, we assume the additional assumptions in Section A.2.
Then, by running the algorithm in Section A.1 with the parameters in Table 1, with probability at least 1 − p , the collected data can be used to implement the following:
(a) An ε -suboptimal hierarchy oracle.
(b) A function AvExt( s ) : Ent( S )
→ P (Ext( S )) such that, given s ∈ Ent( Z s ) , returns Ext( Z s ) .
The algorithm achieves both of these with query complexity
[Equation]
A.4. I MPLEMENTING THE H IERARCHY O RACLE
We ﬁrst show that we can implement the hierarchy oracle in this section.
In particular, we have the following result:
Proposition A.4.
Let M be the MDP corresponding to the index ( x, f, r, ˜ H ) as described in Deﬁnition 4.3.
Then, given ( x, f, r, ˜ H ) , we can form the following estimator for P f :
[Equation]
where ˆ P 0 is the estimator obtained from Phase II in Section A.1.
Assuming that the high-probability event in Proposition A.2 holds for δ ≤ ρε/ 2 SH 2 , value iteration using ˆ P f returns a policy π such that V M , ∗( x ) − V M ,π( x 0 ) ≤ ε .
Throughout the rest of this section, we ﬁx the tuple ( x, f, r, ˜ H ) and the corresponding MDP M .
Furthermore, we write Z for the cluster containing x .
To prove Proposition A.4, we will show that M can be sufﬁciently simulated so that the value of any policy can be reasonably estimated.
Given this simulation result, we can then show that value iteration ﬁnds the desired policy.
This simulation result depends on the following intermediate result, which provides insight as to why Phase II data is sufﬁcient:
Lemma A.13.
For any s ∗ ∈ Z ,
[Equation]
Proof.
First, we note that there exists an MDP such that P M 1 (2 H ) ( s ∗ ∈ τ π ) is the corresponding value function.
In particular, modifying M 1 (2 H ) so that any action from s ∗ leads to a terminal state and deﬁning r ( s, a ) =
[ s = s ∗ ] results in such an MDP.
Now, let π x and π s ∗ be the policies achieving
[Equation]
respectively.
Consider the concatenation of π x and π s ∗ into a history-dependent policy that runs π x until the agent reaches s , and switches to π s ∗ thereafter.
This policy reaches s with probability at least
[Equation]
within the modiﬁed MDP described above.
Since the optimal value among all policies is achieved by a history-independent policy, we obtain the desired inequality.
Informally, the prior result states that if x is reachable within horizon H , then any state reachable from x within Z is also reachable in M 1 within a 2 H horizon.
Therefore, performing rewardfree RL with horizon 2 H during Phase II provides coverage over all clusters.
Now, we prove the simulation result.
Lemma A.14.
Assume that the Phase II guarantee in Proposition A.2 is instantiated for δ ≤ ρε/ 4 SH 2 .
Then, if V π is the value of π under M , and ˆ V π is its corresponding estimate under ˆ P f , then
[Equation]
Proof.
The proof follows similarly to that of Lemma A.7.
By the performance difference lemma,
[Equation]
Observe that if s ∈ S \ Z , P πh ( s, a ) = 0 for any π .
Furthermore, since the dynamics within {(cid:9) S , (cid:9) F } are known, (ˆ P f − P f )
ˆ V πh +
( s, a )
= 0 for s ∈ {(cid:9) S , (cid:9) F } .
Therefore, we can restrict the sum to be over Z × A .
Now, let Z δ denote the set of δ -signiﬁcant ( s, a ) pairs in Z ×A from x , for some δ to be determined.
For a ﬁxed h ∈ [ ˜ H ] , we can decompose the inner sum as (cid:88) [Equation] a
[Equation]
Bounding (II): Error from δ -Insigniﬁcance
By the deﬁnition of δ -signiﬁcance,
[Equation]
where the last inequality follows from setting δ = ε/ 4 SH 2 .
Bounding (I): Reference Dynamics Error.
By the Cauchy-Schwarz inequality,
[Equation]
Then, (cid:88)
[Equation]
Since x is ρ -signiﬁcant in M 1 (2 H ) by Deﬁnition A.
[Equation]
Lemma A.13 together with δ -signiﬁcance in M implies ρδ -signiﬁcance in M 1 (2 H ) .
Therefore,
[Equation]
where the last inequality follows by part (a) of the Phase II guarantee in Proposition A
[Equation]
Substituting into the prior expression,
[Equation]
Thus by applying part (b) of the Phase II guarantee in Proposition A.2, we have that
[Equation]
Concluding.
By combining the bounds on (I) and (II), we obtain the desired result.
With this estimation result [Equation] we can now prove Proposition A.
[Equation]
Proof of Proposition A.4.
Let π be the policy found by value iteration using ˆ P f , which achieves the maximal value in the corresponding MDP.
Then, by Lemma A.
[Equation]
A.4. D ETERMINING A VAILABLE E XITS
In this section, we prove that we can determine the set of available exits.
We have the following formal result:
Proposition A.5.
Assume access to the ε -suboptimal hierarchy oracle from the previous section and that the guarantee in Theorem A.1 holds.
Then, we can implement the function AvExt( s ) :
Ent( S ) → P (Ext( S )) which, given s ∈ Ent( Z s ) , returns the subset of Ext( Z s ) that is reachable from s .
Proof.
Fix an input x ∈ Ent( S ) , which we assume belongs to some cluster Z x .
It sufﬁces to demonstrate that we can implement
[ e ∈ Z x ] for any ﬁxed e ∈ Ent( S ) .
Deﬁne f
e
( s, a ) =
(cid:26) (cid:9) S ( s, a ) =
e (cid:9) F otherwise and r e ( s, a ) =
[( s, a ) =
e ] .
By Deﬁnition A.3, the MDP M corresponding to the tuple ( x, f e , r e , H ) has optimal value V ∗ =
ε 0
[ e ∈ Z x ∧ e reachable from x ] .
Additionally, by Lemma A.14, | V π 0 ( s ) − ˆ V π 0 ( x ) | ≤ ε/ 2 .
We now proceed by cases.
If e (cid:54)∈ Z x or e is not reachable from x , then V π 0 ( x ) = 0 for any policy π , and thus value iteration can only ﬁnd a policy π with ˆ V π 0 ( x ) ≤ ε 0 / 3 .
Otherwise, for e ∈ Z x , V ∗ 0 ( x ) = ε 0 , and thus value iteration necessarily must ﬁnd a π with ˆ V π 0 ( x ) ≥ 2 ε 0 / 3 .
Putting these together, if ˆ V is the optimal estimated value in M , then
[Equation]
Note that this is implementable for all e ∈ Ext( S ) (and returns a subset of Ext( Z x ) for the query above) since the set of exits are already known.
A.4. F INALIZING THE G UARANTEE : Q UERY C OMPLEXITY
In this section, we ﬁnalize the proof of the meta-training guarantee by computing the query complexity.
Proof of Theorem A.2.
As demonstrated by Proposition A.4 and Proposition A.5, running the algorithm in Section A.1 with the parameters in Table 1 provides the desired guarantees with probability at least 1 − p .
To compute the query complexity, observe that we perform the following number of trajectories while executing the algorithm in Section A.1.
[Equation]
Ignoring terms that do not depend on T or ε , we obtain the claim.
A. B RUTE -F ORCE L EARNING OF THE H IEARCHY
Theorem A.3.
Assume that Algorithm 6 is run with parameters satisfying
[Equation]
and
[Equation]
Then, the set returned by the algorithm is exactly Ext( S ) with probability at least 1 − p .
Furthermore, the algorithm achieves this result with query complexity
[Equation]
2: for do 3:
Create MDP M st so P ( (cid:9) | s a ) = 1 for any a . 4: ˜ r s ( s (cid:48) , a (cid:48) ) ← 1 [ s (cid:48)
=
s ] . 5: Ψ st ← E ULER M st ) 6: for all a
A do 7:
Modify to
8: for
] do 9:
Sample
Play collect is encountered 11:
N s a
← number a
Proof.
For any ( s, a ) ∈ Ext( S ) , Lemma A.2 implies that s is α/H -signiﬁcant for some task t ∈ [ T ] .
Therefore, s is α/CH -signiﬁcant for any task t ∈
[ T ] , by Assumption A.1.
Now, by an argument similar to that used in the proof of Lemma A.2, we have that with probability at least 1 − p/ 3 T , the choice of N E ULER implies
[Equation]
for any exit ( s, a ) ∈ Ext( S ) and a ﬁxed task t ∈ [ T ] .
Therefore, by a union-bound over the tasks, the same guarantee holds for all tasks simultaneously with probability at least 1 − p/ 3 .
Now, for any ﬁxed ( α/CH ) -signiﬁcant ( s, a ) pair, sampling from Ψ st at least N times guarantees that with probability at least 1 − p/ 3 SAT , N t ( s, a ) ≥ N thresh .
Therefore, once again performing the necessary union-bound, we obtain the same result uniformly over any ( α/CH ) -signiﬁcant
( s, a ) and t ∈ [ T ] with probability at least 1 − p/ 3 .
Finally, for a ﬁxed ( s, a ) and t , the estimator for P t ( · | s, a ) satisﬁes the property
that when
N ( s, a ) > 0 ,
[Equation]
with probability at least 1 − p/ 3 SAT , using an argument similar to that used in Lemma A.16.
Again, by a union bound, the same guarantee holds for any ( s, a ) and t ∈
[ T ] .
In particular, for any ( s, a ) with N t ( s, a ) ≥ N thresh ,
[Equation]
Therefore, by a similar argument to Lemma A.10, the following are true:
(a) If ( s
, a ) ∈
Ext( S ) , then there exists t, t (cid:48) for which
[Equation]
(b) If ( s, a ) (cid:54)∈ Ext( S ) , then for any t (cid:54) = t (cid:48) with N t ( s, a ) , N t (cid:48) ( s, a ) ≥ N thresh ,
[Equation]
Putting everything together, we see that the set returned by Algorithm 6 is exactly Ext( S ) , with probability at least 1 − p .
A. T ECHNICAL L EMMAS
Lemma A.15.
Let X 1 , . . .
X M be i.i.d.
Ber ( p ) random variables.
Then, if
[Equation] then with probability at least 1 − δ ,
[-M (cid:88) i = 
 [ X i = 1] ≥ N.-][Equation]
Proof.
By applying Hoeffding’s inequality,
[Equation]
Setting the ﬁnal expression to the failure probability δ and solving, we obtain the quadratic inequality
[Equation]
Finally, via solving this inequality for M , we ﬁnd that
[Equation]
is sufﬁcient to guarantee the desired event with failure probability δ , as desired.
Lemma A.16 (Dynamics estimation error bound) .
Fix a policy π , MDP with stationary dynamics
M = ( S , A , P , r, H ) , and N ∈ N .
Assume that π is played N times in M , and all transitions are used to form an estimator ˆ P ( · | s, a ) using empirical averages.
For any ( s, a ) ∈ S × A , let N ( s, a ) be the number of times ( s, a ) is encountered in this process.
Then, with probability at least 1 − p , any ( s, a ) with N ( s, a ) > 0 satisﬁes
[Equation]
Proof.
Assume that the obtained samples are given by { ( s k , a k , s (cid:48) k )
| k ∈ [ HN ] } , so that ( s Hn + r , a Hn + r , s (cid:48)
Hn + r +1 ) is the r th time step in the n th execution of π in M for any 0 ≤ n ≤ N −and
≤ r ≤ H − 1 .
Fix any ( s, a ) ∈ S × A , and assume that ( s ( j ) , a ( j ) , s (cid:48) ( j ) ) is the j th sample from P ( · | s, a ) .
Furthermore, let m j
( s, a ) denote the index at which the j th sample is obtained.
We claim that for any s ∗ ∈ S and 0 < M ≤ HT ,
[Equation]
Let F i be deﬁned as the σ -algebra induced by the set of random variables
[Equation]
Clearly, ( F i ) is a ﬁltration such that the j th term in the sum above is measurable with respect to F j .
Furthermore, observe that
[Equation]
Therefore [Equation] the random variables in the sum forms martingale difference sequence.
Furthermore, the sequence is bounded in [ − 1 , 1]
[Equation]
and satisﬁes
[Equation]
Therefore, by applying Azuma-Bernstein
[Equation]
we have that
[Equation]
with probability at least 1 − p/SAHN .
By applying a union bound on ( s, a, s ∗ ) and M , we thus have that with probability at least 1 − p ,
[Equation]
holds for any ( s, a, s ∗ ) and M .
Conditioned on this event, we thus have that for any ( s, a ) with
N ( s, a ) > 0 ,
[Equation]
ˆ P t
[Equation]
· | s, a ) − P
[Equation]
a
[Equation]
TV
[Equation]
| s, a ) (cid:12) (cid:12)(cid:12) (cid:46) (cid:88) s (cid:48) ∈S (cid:115) P ( s (cid:48) | s, a )
N ( s, a ) log
SAHN δ +
S
N ( s, a ) log
SAHN δ (cid:46) (cid:115)
S
N ( s, a ) log
SAHN δ +
S N ( s, a ) log SAHN δ . The ﬁnal result follows simply by noting that
[Equation]
Lemma A.17.
Fix two MDPs M =
( S , A , P , r, H ) and M (cid:48) =
( S , A , P (cid:48) , r, H ) .
Let ∆ denote the subset of S
×
A ×
[ H ] for which P h ( · | s, a ) (cid:54) =
P (cid:48) h ( · | s, a ) .
Then, for any policy π ,
V M (cid:48) ,π( s 0 ) − V M , ∗( s 0 ) >
ρ = ⇒ P M ( τ π ∩ ∆ (cid:54) = ∅ ) = P M (cid:48) ( τ π ∩ ∆ (cid:54) = ∅ ) > ρ H .
Proof.
Write q = P M (cid:48) ( τ [Equation]
∩ ∆ (cid:54) = ∅ ) . Note that V M (cid:48) [Equation] s 0 ) can be decomposed as
[Equation]
Since P and P (cid:48) agree on ( S × A ×
[ H ]) \ ∆ , the dynamics of M and M (cid:48) agree up until π performs an action in ∆ , and thus [Equation]
[Equation]
Furthermore,
[Equation]
Putting everything together,
[Equation]
A. W HY S HOULD O PTIMISTIC I MAGINATION H OLD ?
In this section, we provide a heuristic explanation as for why optimistic imagination can be expected to hold in most scenarios for which Deﬁnition 3.1 holds.
Recall from the main text that if an exit e satisﬁes the preliminary coverage condition, then e can be detected using only Phase I data.
Therefore, we can restrict our discussion of Assumption 4.1 to subsets of exits that only have nonzero importance for a task t if P t ( · | e ) is some distribution d e .
In other words, we can only guarantee that near-optimal policies for the source tasks observe d e .
We will provide a heuristic reason as to why we can then expect to ﬁnd an MDP such that condition (b) of Assumption 4.1 holds for such a subset S .
State visitation measures.
Recall that the value function in an MDP with a stationary reward function is given by
[Equation]
Now, if we assume that the reward is only a function of the state (as in goal-conditioned RL), and view r as a vector in R |S| , then we can write V M ,π
[Equation]
where
[Equation]
That is, ρ M ,π is the state visitation measure of π within M with horizon H .
This geometric viewpoint was explored by Eysenbach et al.
(2021) in the context of unsupervised RL in the discounted inﬁnite-horizon setting.
We note that for any ﬁxed M , { ρ M ,π } is a convex subset of the probability simplex.
Constructing the desired reward function.
Let M be any source task such that the set T = (cid:8) e ∈ S (cid:12)(cid:12) P M ( · | e ) (cid:54) =
d e (cid:9) is non-empty.
Furthermore, let ˜ M be the corresponding MDP described in Assumption 4.1(b) after borrowing dynamics, so that
[Equation]
otherwise is the dynamics of ˜ M .
Since (cid:13)(cid:13) P M ( · | e ) − d e (cid:13)(cid:13) TV > β for any e ∈ S , heuristically, we expect the following to be true:
Assumption A.2.
There exists a π ∗ such that
ρ ˜ M
[Equation]
π
[Equation]
Stated less formally, we expect there to be states that π ∗ can reach better in ˜ M compared to any policy in M .
Example A.1.
In the four-room example, this intuition is captured by the fact that the agent can spend more time in states past open gates compared to if they were closed.
[Equation]
Under Assumption A.2 above, since { ρ M ,π } is convex, we can ﬁnd a reward direction separating { ρ M ,π } and ρ ˜ M ,π ∗ , or more formally,
[Equation]
Then, the MDP M with reward function r satisﬁes the desired condition in Assumption 4.1(b).
Thus, under Assumption A.2, the coverage condition can indeed hold with sufﬁciently diverse source tasks.
B M ETA -T EST P ROOFS
We now provide an analysis of the regret incurred by a learner using an approximately learned hierarchy at meta-test time.
We ﬁrst show that the hierarchy oracle from the source tasks can provide useful temporally extended behavior.
We then show that using these policies results in bounded suboptimality and achieves a better regret bound compared to standard UCB-VI.
Throughout this section, we ﬁx an optimal π ∗ satisfying the conditions of Assumption 5.1.
Furthermore, we assume that we have access to a hierarchy oracle that provides ε -suboptimal policies as deﬁned in Deﬁnition 4.3.
B. U SING THE H IERARCHY O RACLE
In this section, we show that the hierarchy oracle can be used to implement two useful behaviors: (1) reaching exits and (2) behaving optimally within a cluster.
B.1. N EAR -O PTIMAL G OAL R EACHING
Assume that the agent is currently at a state z ∈ { s 0 }∪ Ent( S ) at time step h , and intends to exit the current cluster Z via exit g = ( s ∗ , a ∗ )
∈
Ext( Z ) .
We obtain a policy implementing the high-level intent as follows:
(1) Deﬁne the termination for any ( s, a
) ∈
Ext( S ) as: f g ( s, a ) := (cid:26) (cid:9) S ( s, a ) = g (cid:9) F otherwise (2) Deﬁne reward as r (cid:9) S ( s, a ) :
=
[ s = (cid:9) S ] (3) Provide ( z, f g , r (cid:9) S , H − h ) to the hierarchy oracle and obtain a policy π z,g,h .
For simplicity, we will write T hier H − h
( z, g ) for T π z,g,h H − h
( z, g ) throughout our analysis.
The following proposition quantiﬁes the performance of the obtained policy:
Proposition B.1.
T hier satisﬁes the following inequality:
[Equation]
Proof.
Due to the deﬁnition of P f g and r
[Equation]
observe that for any π ,
[Equation]
Therefore,
[Equation]
B.1. N EAR -O PTIMAL W ITHIN -C LUSTER B EHAVIOR
Assume that the agent is currently at a state z ∈ { s 0 } ∪ Ent( S ) at time h
[Equation]
and intends to remain in the current cluster Z while maximizing a given reward function r .
We obtain a policy for this high-level intent as follows:
(1) Deﬁne transition dynamics for any ( s, a
) ∈
Ext( Z ) as P ( · | s, a ) =
δ ( (cid:9) F ) .
(2) Provide P , r, and planning horizon H − h to the hierarchy oracle, and obtain a policy π .
B. F ORMAL L EARNING P ROCEDURE
In this section, we describe the procedure for learning a policy using the oracle-provided policies described in the previous section.
Formally, we construct a surrogate MDP whose dynamics are determined by M and the oracle.
We can then apply any tabular learning method to this new MDP (in our case, E ULER ), obtaining a policy in the surrogate MDP that readily translates into a policy in M .
The components deﬁning the surrogate M hl =
( Z , G , P hl , R hl , H eﬀ ) are as follows:
Meta-state space Z .
We set
[Equation]
where ¯ H is a high-probability bound on the time to move through H eﬀ exits (to be determined later).
We incorporate the time step into the meta-state to ensure that both the dynamics and reward are computable from the state information (ensuring that M hl is indeed an MDP).
Meta-action space G .
Given a current meta-state ( s, h ) where s ∈ Z , the available meta-actions
G can be identiﬁed with Ext( Z ) ∪ {(cid:9)} .
Algorithm
1: procedure P ERFORM M ETA T RANSITION ( ( z, g ) ∈
Z × G ) (cid:46) Executes the desired meta-transition in the original MDP M .
2: if z = (cid:9) then 3: return (cid:9) 4: else if z = ( s, h ) then 5: if h ≤ ¯ H then 6: if s ∈ Z ∗ or g = (cid:9) then 7:
Execute within-cluster policy from oracle until termination.
8: return (cid:9) 9: else 10:
Execute π z,g,h obtained from oracle until g is performed or h = ¯ H .
11: s (cid:48) , h (cid:48) ← current state and time step 12: if g was performed then 13: return ( s (cid:48) , h (cid:48) )
14: else 15: return ( s, ¯ H + 1) 16: else 17: if s ∈ Z ∗ or g = (cid:9) then 18: return (cid:9) 19: else 20: return ( s, h )
Meta-dynamics P hl .
Fix ( z, g ) ∈
Z × G for some z (cid:54) = (cid:9) , so that z = ( s, h ) .
We consider the procedure in Algorithm 7 for generating the meta-dynamics.
Intuitively, we execute a meta-action g (cid:54) =
(cid:9) by running the oracle-provided policy until the learner encounters g , or has acted for ¯ H timesteps in the current episode.
On the other hand, if g = (cid:9) , the agent executes the oracle-provided ε -suboptimal policy that remains within the current cluster and acts for H − h timesteps.
Formally, the next state z (cid:48) is given by z (cid:48) =
  (cid:9) s ∈ Z ∗ or g = (cid:9) ( s (cid:48) , h (cid:48) )
h ≤ ¯ H ( s, h ) otherwise , where s (cid:48) and h (cid:48) are generated given T hier H − h ( s, g ) as
[Equation]
| T
[Equation]
s
[Equation]
g )
=
min
[Equation]
T
[Equation]
s
[Equation]
otherwise .
Note that the learner can only execute meta-actions while h ≤ ¯ H .
Furthermore, given access to M , one can easily simulate the dynamics of M hl .
Meta-reward R hl .
Fix (( s, h ) , g
) ∈
Z × G .
Recall that the reward function of M is supported on Ext( S ) ∪ ( Z ∗ ) ◦ .
Thus, this reward function can be lifted onto M hl .
Formally, we deﬁne the following reward function:
[Equation]
otherwise
[Equation]
where W h ( s ) is the random sum of rewards obtained by playing a within-cluster policy starting from s (cid:48) for the rest of the episode.
Note that R hl depends on P hl and is thus random.
Furthermore, this reward function is consistent with how meta-transitions are performed in Algorithm 7.
Meta-horizon H eﬀ .
Recall that there exists an optimal policy that encounters at most H eﬀ exits with high probability.
Accordingly, we limit the learner to being able to choose H eﬀ high-level actions, which recall can be choices of exits.
Solving M hl .
To obtain the desired policy, we apply E ULER to M hl .
By the construction in
Algorithm 7, the policy set returned by E ULER easily translates into policies on M .
Furthermore, the value of this policy is the same on both MDPs.
[Equation]
Having deﬁned the procedure for learning a policy using the hierarchy, we now proceed with the regret analysis.
Our analysis proceeds by constructing a policy expressible in M hl that achieves near-optimal returns by imitating the high-level decisions made by π ∗ .
We then use this policy as a comparator policy when applying E ULER regret bounds to M hl .
To formally construct the desired comparator policy, we need to ﬁrst deﬁne the notion of a metahistory, which contains the set of high-level decisions made by any policy:
Deﬁnition B.1.
Fix a policy π , which given some horizon L , generates a (random) trajectory ( s 0 , a 0 , . . .
, s L ) .
Let Ext( π ) be the number of exits performed in the trajectory, i.e.
[Equation]
The meta-history H hl ( π ) corresponding to this trajectory is the sequence (
[Equation]
0 , g 0 , z
[Equation]
g
[Equation] [Equation]
[Equation] [Equation]
) , s
[Equation]
( s
[Equation]
, s i Ext( π ) ) , where i n :=
[Equation]
otherwise
[Equation]
Note that z i ∈ Ent( S ) and
g i ∈
Ext( S ) for all i = 0 , . . .
, Ext( π ) .
We omit π in writing H hl when the underlying policy π is understood.
[Equation]
Informally, H hl tracks all entrances and exits contained in a trajectory generated by π .
We deﬁne the length of a meta-history H hl , denoted as |H hl | , as the number of exits contained in H hl .
B.3. P OLICY C ONSTRUCTION
We now proceed with constructing the desired policy.
Intuitively, the comparator imitates the distribution over H hl ( π ∗ ) , conditioned on |H hl ( π ∗ ) | ≤ H eﬀ .
To see why this is sufﬁcient for nearoptimality, recall that the reward on M Tg is supported on Ext( Z ∗ ) ∪ ( Z ∗ ) ◦ .
Consequently, by imitating the distribution over meta-histories, the policy is expected to obtain roughly the same sum of rewards in expectation from the exits.
Therefore, all that remains is to ensure that the learner collects roughly the same sum of rewards from Z ∗ , which is the same as ensuring that this policy does not take too long to reach Z ∗ .
Construction.
Let H be the running meta-history, containing k ≤ H eﬀ actions.
The optimal policy induces a distribution q ( · | H ) over A hl representing the next exit it takes 6 .
We then deﬁne π as
[Equation]
otherwise .
Observe that π terminates the episode upon reaching ¯ H .
Furthermore, this policy is dependent on the meta-history.
However, since M hl is an MDP, there exists a stationary policy that achieves at least the same value.
B.3. S UBOPTIMALITY A NALYSIS
In this section, we prove that π achieves bounded suboptimality.
Rather than analyzing π directly in M hl , we construct a new ˜ M hl and ˜ π to better track the meta-history.
In particular, conditioned on the event that π requires more than ¯ H time steps to execute, then the agent would not be able to imitate the full meta-history generated by π ∗ , even after having performed less than H eﬀ exits.
Constructing a surrogate for analysis.
We now formalize the construction of the surrogate MDP ˜
M hl and the policy ˜ π corresponding to π in this MDP.
To obtain ˜ M hl , we redeﬁne the dynamics from M hl so that s (cid:48) | h (cid:48) ∼ P ( · | g ) in ˜ M hl .
In effect, we allow the policy to continue performing transitions beyond ¯ H , although without any reward.
Accordingly, we deﬁne ˜ π
as ˜ π ( · | z, H ) =
q ( · | H ) .
The following lemma formalizes how ˜ π and ˜ M hl have desirable properties for the analysis:
Lemma B.1 (Surrogate Policy Characterization) .
Let µ ∗ denote the distribution of
H hl ( π ∗ ) | |H
hl ( π ∗ ) | ≤ H eﬀ in M Tg , and ˜ µ the distribution of H (˜ π ) in ˜ M hl .
Then, (1 − ζ ) µ ∗ ≤ ˜ µ .
Proof.
Let ν ∗ be the distribution induced by the following procedure:
(1) Sample a meta-history from the distribution H hl ( π ∗ ) | |H
hl ( π ∗ ) | > H eﬀ .
(2) Truncate the obtained meta-history to length H eﬀ .
It is easy to see from the deﬁnition of ˜ π that ˜ µ
= (1 − ζ ) µ ∗ + ζν ∗ .
The desired result follows.
Thus, we have indeed shown the desired property that ˜ π properly tracks the (truncated) meta-history generated by π ∗ .
To justify performing our analysis on ( ˜ M hl , ˜ π ) , we have the following result, which shows that any result on the value of the pair above applies to the value of π in M hl .
Lemma B.2.
As constructed above, V ˜ M hl , ˜ π
[Equation]
Proof.
We write M :=
M hl and M (cid:48) :
=
˜ M hl .
Similarly, we write π (cid:48) :
=
˜ π .
We proceed by proving a chain of equalities.
( V π (cid:48) , M (cid:48) ( s 0 ) =
V π, M (cid:48) ( s 0 ) ).
We omit M (cid:48) in this part of the argument for clarity.
By the performance difference lemma, we have that for any k ∈
[ H eﬀ ] and z ∈ S hl ,
[Equation]
Let ∆ :=
(cid:8) z ∈ S hl (cid:12)(cid:12)
z
= ( s, h ) , s (cid:54)∈ Z ∗ , h ≥ ¯ H (cid:9) , which is the set on which π and π (cid:48) disagree.
Observe that for any π and k , V πk ( z ) = 0 for any z ∈ ∆ , and thus A π (cid:48) k
( z, π )
=
for all such states.
For any other z , A π (cid:48)
k ( z, π ) is clearly 0 .
Thus, we obtain the desired result.
( V π, M (cid:48) ( s 0 )
=
V π, M ( s 0 ) )
We omit π in this part of the argument for clarity.
Using the simulation lemma,
[Equation]
Observe that the behavior of the two MDPs are identical conditioned on h (cid:48) ≤ ¯ H .
On the other hand, conditioned on h (cid:48) > ¯ H , π can no longer receive rewards from either MDP.
Therefore, [( P M − P
M (cid:48) )
V M j ]( z, g ) =
0 for any j, z, g by decomposing the relevant expectations along the two events.
We thus obtain the desired result.
Analyzing the surrogate.
With the results above, we now proceed to analyze the difference in values
[Equation]
which then implies the desired suboptimality result.
First, we have the following lemma characterizing the time ˜ π requires to fully execute a given meta-history in the base MDP M :
Lemma B.3.
Fix any H hl =
( z 0 , g 0 , . . . )
such that |H hl | ≤ H eﬀ .
Furthermore, deﬁne the sequence of reaching times
[Equation]
We deﬁne T hier ( H hl ) to be the time required by the hierarchy to execute H hl , which is formally given by T |H hl | in the sequence above.
Then,
[Equation]
a) E (cid:2) T hier ( H hl )
[Equation]
≤
[Equation]
1 + (1 + γ )
[Equation]
+
[Equation]
eﬀ .
(b) Let σ 2 := β 2 [(1 + γ ) W + ε ] 2 H eﬀ .
Then, for any t > 0 ,
[Equation]
Proof.
We prove the two parts separately:
(a) We will prove via induction that E
[ T k ] ≤ k [1 + (1 + γ ) W + ε ] .
For any k and T k − 1 ,
[Equation]
where the ﬁrst inequality uses properties of the hierarchy oracle, while the ﬁnal inequality follows by combining Deﬁnition 5.2(b) and Assumption 5.2.
Therefore, by linearity and the tower property of expectation,
[Equation]
The desired result then follows by induction.
(b) Let B k :=
k [1 + (1 + γ ) W + ε ] and f k ( t ) :
= E (cid:2) T hier H − t ( z k , g k ) (cid:3) .
Note that for any k and t , B k − 1 + f k − 1 ( t ) ≤ B k , by following the argument in (a).
Therefore, for any λ > 0 ,
E [exp
[Equation]
λ ( T k − B k ) }
[Equation]
where the last inequality uses the monotonicity of the exponential function.
Therefore, by applying the sub-Gaussian condition given in Deﬁnition 5.2,
[Equation]
where we have used the fact that T hier H − T k − 1 ( z k − 1 , s ( g k − 1 )) has a sub-Gaussian upper tail with variance proxy
[Equation]
Note that we have once again used the properties of the hierarchy oracle, and Assumption 5.2.
Therefore, by induction, E [exp { λ ( T k − B k ) } ] ≤ E
[Equation]
λ
[Equation]
√ kC )
2 / 2 (cid:105) , from which the desired tail bound follows by making use of Chernoff’s inequality.
As we have shown that ˜ π closely tracks the meta-history of π ∗ and have analyzed the distribution of time it takes to execute a given meta-history, we can now analyze its suboptimality:
Lemma B.4.
There exists a policy π expressible in M hl such that
[Equation]
Proof.
Assume that π ∗ generates a (random) meta-history of length N given by H hl =
( z 0 , g 0 , z 1 , g 1 , . . .
, z N ) .
Furthermore, let T ∗ denote the (random) time π ∗ takes to reach z N .
Then, given H hl and T ∗ , observe that we can write
[Equation]
where R
[Equation]
g
[Equation]
) , using the assumptions on the reward function and condition (a) in Assumption 5.1.
Subsequently, letting E be the event { N ≤ H eﬀ } , we can bound the right-hand side as
[Equation]
where we have used Assumption 5.1 to bound the probability that N > H eﬀ .
Our goal for the rest of this proof is to transform the expectation on the right-hand side into a form that lower bounds V π 0 ( s 0 ) .
To this end, we deﬁne
[Equation]
and the sequence of times
[Equation]
Note that R hier and T N are analogous to R ∗ and T ∗ , respectively.
Then, letting F be the event (cid:8)
T N ≤ ¯ H (cid:9) , note that
[Equation]
We bound (I) and (II) separately.
Bounding (I) .
Let G be the event E ∩ { z N ∈ Z ∗ } .
Then, if we deﬁne
[Equation]
as the minimum time needed to execute H hl , we then have that
[Equation]
Note that the bound on T min follows from Assumption 5.2.
To convert the different in values into a difference of times, observe that if T N − T min ≤ α − ε , then
[Equation]
Therefore,
[Equation]
where the ﬁnal inequality integrates the tail bound provided in Lemma B.3.
Overall, we have that by rearranging,
[Equation]
Bounding (II) .
By the characterization of ˜ π in Lemma B.1,
[Equation]
where the ﬁnal inequality uses the fact that ¯ R ¯ T ( H hl (˜ π )) is the return of ˜ π in ˜ M hl , given F .
Concluding.
Putting all of the previous bounds together, we ﬁnd that
[Equation]
By setting ¯ H to
[Equation]
sub-Gaussian tail bounds on T N implies that P (cid:0) F C (cid:12)(cid:12) E (cid:1) ≤ ζ .
Finally, by Lemma B.2,
[Equation]
where the last equality follows by the construction of M hl .
We thus obtain the desired suboptimality bound.
B.3. R EGRET A NALYSIS
As earlier suggested, we now make use of ˜ π
as a comparator policy in order to prove a regret bound on a learner making use of the procedure outlined in Section B.2.
Theorem B.1.
Assume that E ULER generates policies π 1 , . .
, π N on M hl , as constructed in Section B.2.
Then, we have the following regret bound:
[Equation]
where
[Equation]
Proof.
Throughout the proof, we consider applying E ULER to M hl where the rewards are scaled by 1 /H to ensure that rewards are bounded in [0 , 1] .
As a result, we can bound G ≤ 1 in the E ULER regret bound in Zanette & Brunskill (2019), since the sum of rewards in M hl is also the sum of rewards in M , and scaling by 1 /H gives the desired bound on G .
Therefore,
N (cid:88)
[Equation]
Furthermore,
[Equation]
We thus obtain the desired result.
B. A N E XPONENTIAL R EGRET S EPARATION FOR A H IERARCHY -O BLIVIOUS L EARNER
In this section, we provide proof of the exponential regret separation between a hierarchical learner and a learner oblivious to the hierarchy.
The overall idea behind our proof is the reduction of solving the family of minimax instances described in Domingues et al.
(2021) to a particular family of task distributions.
B.4. T HE H ARD T ASK D ISTRIBUTION F AMILY
In this section, we describe the family of task distributions that forces any meta-training-oblivious learner to incur exponential regret.
For any string s , we write | s | for its length.
We now deﬁne the family of binary tree room MDPs M W of depth W .
We index a member of this family by a tuple ( (cid:96) ∗ , a ∗ , e ∗ ) , where (cid:96) ∗ is a binary string of length W − 1 , and a ∗ , e ∗ ∈ { 0 , 1 } .
The
MDP M ( (cid:96) ∗ ,a ∗ ,e ∗ ) =
( S , A , P ( (cid:96) ∗ ,a ∗ ,e ∗ ) , r, H ) corresponding to this tuple is constructed as follows:
State Space S .
We create a root state s root , 2 W − 1 states indexed by binary strings of length at most W − 1 collected into a set T = { s 0 , s 1 , s 00 , s 01 , . . .
} , a gate state s gate , and terminal states (cid:9) trap , (cid:9) S , (cid:9) F .
Action Space A .
The set of available actions at every state is the set { 0 , 1 } .
Transition Dynamics P ( (cid:96) ∗ ,a ∗ ,e ∗ ) .
We deﬁne the dynamics as follows:
[Equation]
, b ∼ Ber (1 / 2) bδ ( s gate )
+ (1 − b ) δ ( (cid:9) trap ) s
= s (cid:96) ∗ , b ∼ Ber (1 / 2 + ε 1 [ a = a ∗ ])
δ ( (cid:9) S ) s
= s gate , a = e ∗ δ ( (cid:9) F ) s
= s gate , a (cid:54) =
e ∗ .
Reward Function r .
The reward function is r ( s, a ) =
[ s = (cid:9) S ] + 1 [ s = s gate , a = a ∗ ] .
Having described all the components of every member of M W , all that remains is to construct the family of task distributions T W .
Each member of this family will be indexed by ( (cid:96) ∗ , a ∗ ) , where (cid:96) ∗ and a ∗ are as described above.
Then, the task distribution T ( (cid:96) ∗ ,a ∗ ) ∈
T W chooses uniformly within the set (cid:8) M ( (cid:96) ∗ ,a ∗ , 0) , M ( (cid:96) ∗ ,a ∗ , 1) (cid:9) .
Note that this implicitly deﬁnes the latent hierarchy so that the clusters are { s root , s gate , (cid:9) trap } ∪ T , {(cid:9) S } , and {(cid:9) F } .
Furthermore, the set of exits for the ﬁrst cluster is { ( s gate , 0) , ( s gate , 1) } .
B.4. A F AMILY OF H ARD I NSTANCES
In this section, we describe the family of hard instances which we reduce to solving the task distribution above.
Intuitively, if an algorithm incurs low regret throughout M W , then it must be able to quickly ﬁnd a policy to reliable reach the gate state s gate for any MDP in the family.
Constructing the hard instances.
Accordingly, we deﬁne a new MDP family N W , which now is only indexed by ( (cid:96) ∗ , a ∗ ) , and is constructed similarly as any member of M W , but ignoring states outside { s root , s gate , (cid:9) trap }∪ T .
Additionally, we redeﬁne the reward function r for any member to be r ( s, a ) :
=
[ s = s gate ] .
We note that this is exactly the set of hard tasks used to prove a minimax regret bound in Domingues et al.
The lower bound.
We state the lower bound result from Domingues et al.
(2021), in a slightly more restricted form for ease of proof and presentation.
In particular, we consider the following more restricted deﬁnition of an algorithm:
Deﬁnition B.2.
Let H n be the trajectory data generated by playing a policy π n in an MDP M .
That is, H n =
(( s 0 , a 0 , r 0 , s 1 ) , ( s 1 , a 1 , r 1 , s 2 ) , . .
, ( s H − 1 , a H − 1 , r H − 1 , s H )) , where s 0 and a 0 are ﬁxed, r h
= r
( s h , a h ) , and s h +1 ∼ P M ( · | s h , a h ) .
Additionally, we set H 0 = ∅ .
Then, a valid algorithm A for our purposes is one which, for the n th episode, outputs a deterministic, non-stationary policy π that is solely a function of the current state and action and (cid:83) n − 1 i =1 H i .
That is, A does not output policies that adapt to the current running episode.
[Equation]
We again emphasize that this restriction is not necessary but that many algorithms nevertheless satisfy this condition (including UCBVI and E ULER ).
We then have the following hardness result:
Theorem B.2 (Domingues et al.
(2021), Theorem 9, restated) .
Assume that W ≥ 2 and H ≥ 3 W .
Then, for every algorithm A , there exists an MDP M ∈ N W such that
[Equation]
Algorithm
The reduction of learning N W to learning M W in Section B.4.3.
1: procedure P A ( M ∈ N W ) 2:
Initialize H
= ∅ 3: for all n ∈ [ N ] do 4:
Obtain π n =
A ( H 0 , . . .
, H n − 1 ) .
Play π n in M , get history G n = (( s 0 , a 0 , r 0 , s 1 ) , . .
, ( s H − 1 , a H − 1 , r H − 1 , s H )) .
6: if s W +1 =
s gate then 7: s (cid:48)
W +1 ← s W +1 8: for all h = W + 1 , . . .
, H − 1 do 9: if h =
W then 10: s (cid:48)
h +1 ← (cid:9) S if a h = 1 else (cid:9) F .
11: r (cid:48)
h ← 1 [ a h = 1] 12: else 13: s (cid:48)
h +1 ← s (cid:48) h , r (cid:48)
h ← r h 14:
Replace ( s h , a h , r h , s h +1 ) with ( s (cid:48) h , a h , r (cid:48)
h , s (cid:48)
h +1 ) in G n 15:
H n ← G n .
B.4. P ROVING THE H ARDNESS R ESULT
We now use the hardness result in the previous section to demonstrate that no algorithm can incur sub-exponential regret in W on all tasks in M W .
We do so by proving that an algorithm solving all tasks in M W can be used to construct an algorithm for solving all tasks in N W .
Formally, let A be any algorithm for learning any MDP in M W .
We then construct an algorithm P A for learning any MDP in N W as in Algorithm 8.
Given this reduction, we aim to prove the following result:
Proposition B.2.
For any M ( (cid:96) ∗ ,a ∗ ) ∈ N W , we have that
[Equation]
To prove this result, we ﬁrst prove that P A can simulate M ( (cid:96) ∗ ,a ∗ , 1) :
Lemma B.5.
For any n , the distribution over ( H 0 , . . .
, H n ) induced by running Algorithm 8 over
M ( (cid:96) ∗ ,a ∗ ) ∈ N W is equal to that induced by running A over M ( (cid:96) ∗ ,a ∗ , 1) ∈ M W .
Proof.
We proceed by induction.
The result holds trivially for n = 0 .
Now, assume that the result holds for some n .
We condition on the histories ( H 0 , . . .
, H n )
Then, note that both algorithms play the same policy π n +1 , since P A uses A to obtain the next policy.
As a result, by the construction of M ( (cid:96) ∗ ,a ∗ ) and M ( (cid:96) ∗ ,a ∗ , 1) , the distribution over ( s h , a h , r h , s h +1 ) are equal for h ≤ W .
Furthermore, Lines 6 − 14 simulates the dynamics of M ( (cid:96) ∗ ,a ∗ , 1) conditioned on s W +
= s gate , while conditioned on s W +
= (cid:9) trap , the dynamics of the two MDPs are the same.
Therefore, conditioned on any ( H 0 , . .
, H n ) , the distribution over H n +1 induced by the two algorithms are also the same.
Thus, the claim holds by induction.
Finally, we can prove Proposition B.2.
Proof of Proposition B.2.
Throughout this proof, we omit the starting state s root and the timestepin the value.
We prove the result by induction.
Clearly, the result holds for N = 0 .
Assume that the bound holds for some N .
Then, we have that
[Equation]
where the ﬁnal inequality uses the inductive hypothesis and the tower property of expectation.
Now, recall from Lemma B.5 that
[Equation]
We emphasize that the value functions are still with respect to M ( (cid:96) ∗ ,a ∗ ) .
However, for any policy π output by A ,
[Equation]
Note that the right-hand side is the regret in M ( (cid:96) ∗ , a ∗ , 1) for playing π .
Therefore, since both algorithms play the same policy π N +1 , we thus obtain the desired result by induction.
With Proposition B.2, we can now formally state and prove the separation result:
Theorem B.3.
There exists a task distribution T ( (cid:96) ∗ ,a ∗ ) ∈ T W such that an algorithm A , without access to the meta-training tasks (and thus without access to the hierarchy), incurs expected regret lower bounded as
[Equation]
On the other hand, for any task distribution in the family, the hierarchy-based learner P in Section
B.2, with access to a 0 -suboptimal hierarchy oracle, achieves regret bounded by √ H 2 N with high probability on any sampled task.
Proof.
Fix any algorithm A .
Using Theorem B.2, there exists M ( (cid:96) ∗ ,a ∗ ) such that
[Equation]
Thus, by Proposition B.2,
[Equation]
Note that the proof in Proposition B.2 can be extended for M ( (cid:96) ∗ ,a ∗ , 0) with appropriate modiﬁcations to P A , and thus the same inequality holds.
Consequently,
[Equation]
On the other hand, with access to the 0 -suboptimal hierarchy oracle, observe that the learner only has to plan at timesteps 0 and W +1 , allowing us to obtain tighter bounds (as S hl is smaller than the construction in Section B.2).
Furthermore, the suboptimality of planning with the hierarchy oracle is 0 for any task distribution in the family.
We thus obtain the desired bound.
B. A D ISCUSSION OF D EFINITION 5.
In this section, we discuss why the values deﬁned in Deﬁnition 5.2 control the suboptimality of the hierarchical learner.
In particular, we provide examples of MDPs that satisfy Assumption 5.1, and are thus in a sense tasks that are “compatible with the hierarchy”, but nevertheless force a hierarchybased learner to incur O ( H ) suboptimality.
B.5. ( α, β ) - UNRELIABILITY
Consider the MDP in Figure 9 with horizon H + 2 and two actions a ∗ and a 1 .
The optimal policy chooses a ∗ at every step, achieving a value of H − O (1) , since
[Equation]
Now, assume that the MDP has a latent hierarchy so that the set of exits are given by ( t i , a ) for any i ∈
[ H ] and a ∈ A .
Clearly, the optimal hierarchy-based learner would always choose ( t 0 , a ∗ ) or ( t 0 , a 1 ) as its high-level action.
However, if the agent fails to transition to t 0 at the ﬁrst timestep due to stochasticity, it will go to the end of the chain, back to s 0 and try a ∗ once more.
This is because it already has set a meta-action, and does not replan until an exit is performed .
Thus, the optimal agent on the meta-MDP achieves a value of H/ 2 , and is therefore O ( H ) -suboptimal, even with a 0 -suboptimal hierarchy oracle.
Intuitively, hierarchy-based learners as formulated in Section B.2 fail on the MDP in Figure 9 because such learners commit to a skill until completion.
Thus, when such skills exhibit high variance in completion times, hierarchy-based learners fare worse than other learners which are able to replan based on the current state (e.g., in this case, choose another exit if a ∗ fails to take the agent to the current subgoal).
Thus, ( α, β ) -reliability serves to eliminate such MDPs, ensuring that the skills corresponding to reaching exits are reliable.
ss· · · s H −s H tt· · · t H −t H s ∗ 1 , r = 0 .0 .0 .0 .
B.5. γ - GOAL - REACHING SUBOPTIMALITY
In this section, we show that even when a hierarchy-based learner has access to highly reliable skills as in the previous section, the learner may still incur high hierarchical sub-optimality.
Consider the
MDP in Figure 10, where we focus on a single room for simplicity.
Furthermore, assume that there are two exits, one from l H/ 2 and one from r H/ 2 .
Note that a 0 -suboptimal hierarchy oracle has highly reliable goal-reaching policies for reaching both of these exit states, requiring exactly H/timesteps with no stochasticity.
However, given the values assigned to l H/ 2 and r H/ 2 , the optimal policy would opt to take the state t , which transitions to either state with probability at least 1 / 2 in only two environment steps.
Therefore, the optimal policy achieves an optimal value of H − O (1) .
However, the optimal policy, in having to commit to exactly one of the exits, will achieve a value of H/ 2 , and thus be O ( H ) - suboptimal despite having a perfect hierarchy oracle.
Hierarchy-based learners fail on the MDP in Figure 10 because an optimal policy for goal-reaching does not necessarily reach a goal as quickly as possible.
Thus, γ -goal-reaching suboptimality is a regularity condition that ensures that this is indeed the case.
[Figure]
r
[Figure]
[Equation] [Equation]
t
V ∗ h ( r H/ 2 ) =
H
[Equation]
h V ∗ h ( l H/ 2 ) =
H − h 0 .0 .