L EARNING
R EUSABLE
O PTIONS
FOR
M ULTI -T ASK
R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning (RL) has become an increasingly active area of research in recent years.
Although there are many algorithms that allow an agent to solve tasks efﬁciently, they often ignore the possibility that prior experience related to the task at hand might be available.
For many practical applications, it might be unfeasible for an agent to learn how to solve a task from scratch, given that it is generally a computationally expensive process; however, prior experience could be leveraged to make these problems tractable in practice.
In this paper, we propose a framework for exploiting existing experience by learning reusable options .
We show that after an agent learns policies for solving a small number of problems, we are able to use the trajectories generated from those policies to learn reusable options that allow an agent to quickly learn how to solve novel and related problems.
I NTRODUCTION
Reinforcement learning (RL) techniques have experienced much of their success in simulated environments, such as video games (Mnih et al., 2015) or board games (Silver et al., 2016; Tesauro, 1995).
One of the main reasons why RL has worked so well in these applications is that we are able simulate millions of interactions with the environment in a relatively short period of time.
In many real world applications, however, where the agent interacts with the physical world, it might not be easy to generate such a large number of interactions.
The time and cost associated with training such systems could render RL an unfeasible approach for training in large scale.
As a concrete example, consider training a large number of humanoid robots (agents) to move quickly, as in the Robocup competition (Farchy et al., 2013).
Although the agents have similar dynamics, subtle variations mean that a single policy shared across all agents would not be an effective solution.
Furthermore, learning a policy from scratch for each agent is too data-inefﬁcient to be practical.
As shown by Farchy et al.
, this type of problem can be addressed by leveraging the experience obtained from solving a related task (e.g., walking) to quickly learn a policy for each individual agent that is tailored to a new task (e.g., running).
These situations also occurs in industry, such as robots tasked with sorting items in fulﬁllment centers.
A simple approach, like using PD controllers, would fail to adapt to the forces generated from picking up objects with different weight distributions, causing the arm to drop the objects.
RL is able to mitigate this problem by learning a policy for each arm that is able to make corrections quickly, which is tailored to the robot’s dynamics.
However, training a new policy for each agent would be far too costly to be a practical solution.
In these scenarios, it is possible to use a small number of policies learned a subset of the agents, and then leverage the experience obtained from learning those policies to allow the remaining agents to quickly learn their corresponding policies.
This approach can turn problems that are prohibitively expensive to solve into relatively simple problems.
To make use of prior experience and improve learning on new related problems in RL, several lines of work, which are complementary to each other, have been proposed and are actively being studied.
Transfer learning (Taylor & Stone, 2009) refers to the problem of adapting information acquired while solving one task to another.
One might consider learning a mapping function that allows for a policy learned in one task to be used in a different task (Ammar et al., 2015) or simply learn a mapping of the value function learned in one task to another (Taylor et al., 2007).
These techniques can be quite effective, but are also limited in that they consider mapping information from one source task to another target task.
Another approach to reusing prior knowledge is through meta learning or learning to learn (Schmidhuber, 1995; Schmidhuber et al., 1998).
In the context of RL, the goal under this framework for an agent to be exposed to a number of tasks where it can learn some general behavior that generalizes to new tasks (Finn et al., 2017).
One last technique to leverage prior experience, and the one this paper focuses on, is through temporally extended actions or temporal abstractions (McGovern & Sutton, 1998;
Sutton et al., 1999).
While in the standard RL framework the agent has access to a set of primitive actions (i.e., actions that last for one time step), temporally extended actions allow an agent to execute actions that last for several time-steps.
They introduce a bias in the behavior of the agent which, if appropriate for the problem at hand, results in dramatic improvements in how quickly the agent learns to solve a new task.
A popular representation for temporally extended actions is the options framework (Sutton &
Precup, 1998; Sutton et al., 1999) (formally introduced in the next section), which is the focus of this work.
It has been shown that options learned in a speciﬁc task or set of tasks, can be reused to improve learning on new tasks (Machado et al., 2017; Bacon et al., 2017); however, this often requires knowledge from the user about which options or how many options are appropriate for the type of problems the agent will face.
In this paper, we propose learning reusable options for a set of related tasks with minimal information provided by the user.
Throughout this paper, we refer as (near)-optimal policies to those policies that were learned to solve a particular task, but are not strictly speaking optimal.
We consider the scenario where the agent must solve a large numbers of tasks and show that after learning a (near)-optimal policy for a small number of problems, we can learn an appropriate number of options that facilitates learning in a remaining set of tasks.
To do so, we propose learning a set of options that minimize the expected number of decisions needed to represent trajectories generated from the (near)-optimal policies learned by the agent, while also maximizing the probability of generating those trajectories.
Unlike techniques that learn options to rach bottleneck states (McGovern & Barto, 2001) or states deemed of high value (Machado et al., 2017), our method seeks to learn options that are able to generate trajectories known to perform well.
This does not necessarily lead to learn options that reach states one might consider “interesting”.
B ACKGROUND AND N OTATION
A Markov decision process (MDP) is a tuple, M =
( S , A , P, R, γ, d 0 ) , where S is the set of possible states of the environment, A is the set of possible actions that the agent can take, P ( s, a, s (cid:48) ) is the probability that the environment will transition to state s (cid:48) ∈ S if the agent executes action a ∈ A in state s ∈ S , R
( s, a, s (cid:48) ) is the expected reward received after taking action a in state s and transitioning to state s (cid:48) , d 0 is the initial state distribution, and γ ∈ [0 , 1] is a discount factor for rewards received in the future.
We use t to index the time-step and write S t , A t , and R t to denote the state, action, and reward at time t .
A policy , π : S ×
A → [0 , 1] , provides a conditional distribution over actions given each possible state:
π ( s, a ) =
Pr( A t = a | S t = s ) .
We denote a trajectory of length t as h t =
( s 0 , a 0 , r 0 , . . .
, s t − 1 , a t − 1 , r t − 1 , s t ) , that is, h t is deﬁned as a sequence of states, actions and rewards observed after following some policy for t time-steps.
This work focuses on learning options that can be used for a set of related tasks.
We consider the setting where an agent must solve a set of related tasks, where each task is an MDP, M = ( S , A , P M , R M , γ, d M 0 ) ; that is, each task is an MDP with its own transition function, reward function and initial state distribution, with shared state and action sets.
An option, o =
( I o , µ o , β o ) , is a tuple in which I o ⊆
S is the set of states in which option o can be executed (the initiation set ), µ o is a policy that governs the behavior of the agent while executing o , and β o : S → [0 , 1] is a termination function that determines the probability that o terminates in a given state.
We assume that I o =
S for all options o ; that is, the options are available at every state.
The options framework does not dictate how an agent should choose between available options or how options should be discovered.
A common approach to selecting between options is to a learn a policy over options , which is deﬁned by the probability of choosing an option in a particular state.
Two recent popular approaches to option discovery are eigenoptions (Machado et al., 2017) and the option-critic architecture (Bacon et al., 2017).
The eigenoptions (Machado et al., 2017) of an MDP are the optimal policies for a set of implicitly deﬁned reward functions called eigenpurposes .
Eigenpurposes are deﬁned in terms of proto-value functions (Mahadevan, 2005), which are in turn derived from the eigenvectors of a modiﬁed adjacency matrix over states for the MDP.
The intuition is that no matter the true reward function, the eigenoptions allow an agent to quickly traverse the transition graph, resulting in better exploration of the state space and faster learning.
However, there are two major downsides: 1) the adjacency matrix is often not known a priori , and may be difﬁcult to construct for large MDPs, and 2) for each eigenpurpose, constructing the corresponding eigenoption requires solving a new MDP.
The option-critic architecture (Bacon et al., 2017) is a more direct approach to learn options and a policy over options simultaneously using policy gradient methods.
One issue that often arises within this framework is that the termination functions of the learned options tend to collapse to “always terminate”.
In a later publication, the authors built on this work to consider the case where there is a cost associated with switching options (Harb et al., 2018).
This method resulted in the agent learning to use a single option while it was appropriate and terminate when an option switch was needed, allowing it to discover improved policies for a particular task.
The authors argue that minimizing the use of the policy over options may be desirable, as the cost of choosing an option may be greater than the cost of choosing a primitive action when using an option.
Recent work by Harutyunyan et al.
(2019) approaches the aforementioned termination problem by explicitly optimizing the termination function of options to focus on small regions of the state space.
However, in contrast to the work presented in these paper, these methods do not explicitly take into consideration that the agent might face many related tasks in the future.
We build on the idea that minimizing the number of decisions made by an agent leads to the discovery of general reusable options, and propose an ofﬂine method where they are learned by solving a small number of tasks.
The options are then leveraged to quickly solve new problems the agent will face in the future.
We use the trajectories generated while learning (near)-optimal policies, and learn an appropriate set of options by directly minimizing the expected number of decisions the agent makes while simultaneously maximizing the probability of generating the observed trajectories.
L EARNING R EUSABLE O PTIONS FROM E XPERIENCE
In this section, we introduce the objective for learning a set of reusable options for a set of related tasks.
Our algorithm introduces one option at a time until introducing a new option does not improve the objective further.
This procedure results in a natural way of learning an adequate number of options without having to predeﬁne it; a new option is included if it is able to improve the probability of generating optimal behavior while minimizing the number of decisions made by the agent.
Our method assumes that the agent has learn a policy for a small number of tasks, and sample trajectories are obtained from these (near)-optimal policies.
Notice that the propose algorithm is only concerned with being able to recreate the demonstrated trajectories, so if these were sample from a poorly performing policy the options learned are unlikely to provide any beneﬁts.
3. P ROBLEM F ORMULATION
In the options framework, at each time-step, t , the agent chooses an action, A t , based on the current option, O t .
Let T t be a Bernoulli random variable, where T t
= 1 if the previous option, O t − 1 , terminated at time t , and T t = 0 otherwise.
If T t = 1 , O t is chosen using the policy over options, π .
If T t = 0 , then the previous option continues, that is, O t =
O t − 1 .
To ensure we can represent any trajectory, we consider primitive actions to be options which always select one speciﬁc action and then terminate; that is, for an option, o , corresponding to a primitive, a , for all s ∈ S , the termination function would be given by β o ( s ) = 1 , and the policy by µ ( s, a (cid:48) )
=
if a (cid:48) = a and 0 otherwise.
Let O =
O A ∪ O O denote a set of options, { o 1 , . . .
, o n } , where O A refers to the set of options corresponding to primitive actions and O O to the set corresponding to temporal abstractions.
Furthermore, let H be a random variable denoting a trajectory of length | H | generated by a near-optimal policy, and let H t be a random variable denoting the sub-trajectory of H up to the state encountered at time-step t .
We seek to ﬁnd a set, O ∗ =
{ o ∗ 1 , . . .
, o ∗ n } , that maximizes the following objective:
[Equation]
where g ( h, O O ) is a regularizer that encourages a diverse set of options, and λ 1 is a scalar hyperparameter.
If we are also free to learn the parameters of π , then O ∗ ∈ arg max O
[Equation]
One choice for g is the average KL divergence on a given trajectory over the set of m options being learned: g ( h, O O ) =
[Equation]
[Equation]
Intuitively, we seek to ﬁnd options that are capable of generating near-optimal trajectories with a small number of terminations.
Notice that minimizing the number of terminations is the same as minimizing the number of decisions made by the policy over options, as each termination requires the policy to choose a new option.
Given a set of options, a policy over options, and a near-optimal sample trajectory, we can calculate the joint probability for a trajectory exactly , and estimate equation 1 by averaging over a set of near-optimal trajectories.
3. O PTIMIZATION O BJECTIVE FOR L EARNING O PTIONS
Given that the agent must solve a set of tasks, we can use the experienced gathered on a subset of tasks to obtain trajectories demonstrating optimal behavior.
Given a set, H , of trajectories generated from an initial subset of tasks, we can now estimate the expectation in equation 1 to learn options that can be leveraged in the remaining problems.
Because the probability of generating any trajectory approaches 0 as the length of the trajectory increases, we make modify the original objective for better numerical stability, and arrive to the objective ˆ J that we optimize in practice.
[Equation]
A more detailed discussion on how we arrived to this objective is provided in Appendix A. We can express equation 2 entirely in terms of the policy over options π , options O = { o 1 , . . .
, o n } and the transition function, P [-. 
-](which we estimate from samples).
The following theorems show how to calculate the ﬁrst two terms in equation 2, allowing us to maximize the proposed objective.
Theorem 1.
Given a set of options, O , and a policy, π , over options, the expected number of terminations for a trajectory h is given by:
[Equation]
and Pr( O
= o | H
= h 0 , π, O )
=
π ( s 0 , o ) .
Proof.
See Appendix B.
Theorem 2.
Given a set of options O and a policy π over options, the probability of generating a trajectory h of length | h | is given by:
[Equation]
where f is a recursive function deﬁned as:
[Equation]
otherwise
Proof.
See Appendix C.
1: Collect set of trajectories H 2: Initialize option set O with primitive options 3: done = false 4: ˆ J prev =
−∞ 5: while done == false do 6:
Initialize new option o (cid:48)
= ( µ (cid:48) φ , β (cid:48) ψ ) , initializing parameters for φ and ψ .
O (cid:48)
=
O ∪ o (cid:48) 8:
Initialize parameters θ of policy π θ 9: for k=1,...,N do 10: ˆ J k
= ˆ J
( π θ , O (cid:48) , H ) 11: θ
= θ + α ∂ ˆ J k ∂θ 12: φ = φ + α ∂ ˆ J k ∂φ 13: ψ = ψ + α ∂ ˆ J k ∂ψ 14: if ˆ J N − ˆ J prev < ∆ then 15: done = true 16: else 17:
O = O (cid:48) 18: ˆ J prev =
ˆ J N 19: Return new option set
O
Given a parametric representation of the option policies and termination functions for each o ∈ O and for the policy π over options, we use
Theorems 1 and 2 to differentiate the objective in equation 2 with respect to their parameters and optimize with any numerical optimization technique.
3. L EARNING O PTIONS I NCREMENTALLY
One common issue in option discovery is identifying how many options are needed for a given problem.
Oftentimes this number is predeﬁned by the user based on intuition.
In such a scenario, one could learn options by simply randomly initializing the parameters of a number of options and optimizing the proposed objective in equation 2.
Instead, we propose not only learning options, but also the number of options needed, by the procedure shown in Algorithm 1.
This algorithm introduces one option at a time and optimizes the objective ˆ J with respect to the policy over options π θ , with parameters θ , and the newly introduced option, o (cid:48)
=
( µ (cid:48)
φ , β (cid:48) ψ ) , with parameters φ and ψ , for
N epochs.
Optimizing both o (cid:48) and π θ allows us to estimate how much we can improve ˆ J given that we keep any previously introduced option ﬁxed.
After the new option is trained, we measure how much ˆ J has improved; if it fails to improve above some threshold, ∆ , the procedure terminates.
This results in a natural way of obtaining an appropriate number of options, as options stop being added once a new option no longer improves the ability to represent the demonstrated behavior.
E XPERIMENTAL R ESULTS
This section describes experiments used to evaluate the proposed technique approach.
We show results in the “four rooms” domain to allow us to visualize and understand the options produced by our method, and to show empirically that these options produce a clear improvement in learning.
We use this domain to show that options generated by our method are able to generalize to tasks where the option-critic architecture (Bacon et al., 2017) and eigenoptions (Machado et al., 2017) would fail to do so.
We then extend our experiments to evaluate our technique in a few selected problems from the Atari 2600 emulator provided by OpenAI Gym (Brockman et al., 2016).
These experiments demonstrate that by using the trajectories obtained from solving a small subset of tasks, our approach is able to discover options that signiﬁcantly improve the learning ability of the agent in the tasks it has yet to solve.
For the four room experiment, we assume the transition function was known in advance.
In all ATARI experiments, we estimated the transition functions by ﬁtting the parameters of a linear Gaussian model to all the transitions experienced during training.
4. E XPERIMENTS ON F OUR R OOMS E NVIRONMENT
We tested our approach in the four rooms domain: a gridworld of size 40 × 40 , in which the agent is placed in a start state and needs to reach a goal state.
At each time-step, the agent executes one of four available actions: moving left, right, up or down, and receives a reward of − 1 .
Upon reaching the goal state, the agent receives a reward of +10 .
We generated 30 different task variations by changing the goal and start locations, and collected six sample trajectories from optimal policies learned in six tasks.
We evaluated our method on the remaining 24 tasks.
Figure 1a shows the change in the average expected number of terminations and average probability of generating the observed trajectories while learning options, as new options are introduced and adapted to the sampled trajectories.
Options were learned over the six sampled optimal trajectories and every 50 epochs a new option was introduced.
For every new option, the change in probability of generating the observed trajectories as well as the change in expected number of decisions reaches a plateau after 30 or 40 training epochs.
When a new option is introduced, there is a large jump in the loss because a new policy, π , is initialized arbitrarily to account for the new option set being evaluated.
However, after training the new candidate option, the overall loss improves beyond what it was possible before introducing the new option.
In Figure 1b, we compare the performance of Q-learning on 24 novel test tasks using options discovered by our method (with and without regularization using KL divergence), eigenoptions, and option critic.
We allowed each competing method to learn options from the same six training tasks and, to ensure a fair comparison, we used the original code provided by the authors.
As baselines, we also compare against primitive actions and randomly initialized options.
It might seem surprising that both eigenoptions and the option-critic failed to reach an optimal policy when they were shown to work well in this type of problem; for that we offer the following explanation.
Our implementation of four rooms is deﬁned in a much larger state space than the ones where these methods were originally tested, making each individual room much larger.
Since the options identiﬁed by these methods tend to lead the agent from room to room, it is possible that, once in the correct room, the agent executes an option leading to a different room before it had the opportunity to ﬁnd the goal.
When testing our approach in the smaller version of the four room problem, we found no clear difference in performance of the competing methods.
In this experiment, we set the threshold ∆ for introducing a new option to 10% of ˆ J at the previous iteration and the hyperparameter λ 2 = 100 . 0 .
When adding KL regularization, we set λ 1 = 0 . 001 .
Figure 2 shows a visualization of the policy learned by the agent for a speciﬁc task.
The policy leads the agent to navigate from a speciﬁc location in the bottom-left room to a location in the top-right room in a small “four-room” domain of size 10 × 15 .
The new task to solve is shown in the top-left ﬁgure, while the solution found is shown in the top-right ﬁgure.
The remaining rows of images depict the learned option policies, termination functions, and how they were used in the new task.
The ﬁrst row shows the learned option policies after training, the center row depict the termination functions and the bottom row shows a heat-map depicting where each option is likely to be called.
The ﬁgure shows that while the options are deﬁned over the entire state space, they are only useful in speciﬁc regions—that is, they are specialized.
These options, when used in combination in speciﬁc regions, allow the agent to learn how to solve new problems more efﬁciently.
4. E XPERIMENTS USING A TARI 2600 G AMES
We evaluated the quality of the options learned by our framework in two different Atari 2600 games:
Breakout and Amidar.
We trained the policy over options using A3C (Mnih et al., 2016) with grayscale pixel input.
Options were represented by a two layer convolutional neural network, and were given the previous two frames as input.
In both experiments the task variations consisted in changing the number of frames skipped after taking an action (randomly selected betweenand 10 ), the reward function by scaling the reward with a real number between 0 . 1 and 10 . 0 , and initial state distribution by letting the agent execute between 0 and 20 actions before start it starts learning.
The full implementation details for these experiments are given in Appendix E. Figures 3a and 3b show the performance of the agent as a function of training time in Breakout and Amidar, respectively.
The plots show that given good choices of hyperparameters, the learned options led to a clear improvement in performance during training.
For both domains, we found that λ 2 = 5 ,led to a reasonable trade-off between the ﬁrst two term in ˆ J , and report results with three different regularization values: λ 1 = 0 . ,, λ 1 = 0 .
01 and λ 1 = 0 .
Note that our results do not necessarily show that the options result in a better ﬁnal policy, but they improve exploration early in training and enable the agent to learn more effectively.
Figure 4a depicts the behavior for one of the learned options on Breakout.
The option efﬁciently catches the ball after it bounces off the left wall, and then terminates with high probability before the ball has to be caught again.
Bear in mind that the option remains active for many time-steps, signiﬁcantly reducing the number of decisions made by the policy over options.
However, it does not maintain control for so long that the agent is unable to respond to changing circumstances.
Note that the option is only useful in speciﬁc case; for example, it was not helpful in returning a ball bounced off the right wall.
That is to say, the option specialized in a speciﬁc sub-task within the larger problem:
[Figure]
a highly desirable property for generally useful options.
Figure 4b shows the selection of two of the options learned for Amidar when starting a new game.
At the beginning of the game, option 1 is selected, which takes the agent to a speciﬁc intersection before terminating.
The agent then selects option 2, which chooses a direction at the intersection, follows the resulting path, and terminates at the next intersection.
Note that the agent does not need to repeatedly select primitive actions in order to simply follow a previously chosen path.
Having access to these types of options enables an agent to easily replicate known good behaviors, allowing for faster and more meaningful exploration of the state space. [Figure]
C ONCLUSION AND F UTURE W ORK
In this work we presented an optimization objective for learning options ofﬂine from demonstrations of near-optimal behavior on a set of tasks.
Optimizing the objective results in a set of options that allows an agent to reproduce the behavior while minimizing the number of decisions made by the policy over options, which are able to improve the learning ability of the agent on new tasks.
We provided results showing how options adapt to the trajectories provided and showed, through several experiments, that the identiﬁed options are capable of signiﬁcantly improving the learning ability of an agent.
The resulting options encode meaningful abstractions that help the agent interact with and learn from its environment more efﬁciently.
A A PPENDIX
The following list deﬁnes the notation used in all derivations:
A t : random variable denoting action taken at step t .
S t : random variable denoting state at step t .
H t : random variable denoting history up to step t .
H t =
( S 0 , A 0 , S 1 , A 1 , . . .
, S t ) .
T t : random variable denoting the event that the option used at step t − 1 terminates at state S t .
π : policy over options.
P : transition function.
P ( s, a, s (cid:48) ) denotes the probability of transitioning to state s (cid:48) by taking action a in state s 7.
O t : random variable denoting the option selected for execution at state S t . 8.
o : option deﬁned as o =
( µ o , β o ) , where µ o is the option policy for option and β o is the termination function.
Assume primitives are options that perform only 1 action and last for 1 time-step. 10.
O : set of available options.
We can compute the probability of an option terminating at state s t and generating a trajectory h t as:
[Equation]
To compute the proposed objective J we need to ﬁnd an expression for Pr( T t = 1 | H t = h t , π, O ) and Pr( H t = h t | π, O ) in terms of known quantities.
A. A PPENDIX A - D ERIVATION OF ˆ J
Recall J ( π, O , H ) =
E (cid:104)(cid:80)
| h | t =1 Pr( T t = 0 , H t | π, O ) (cid:105) , ignoring the regularization term.
Assuming access to a set H of sample trajectories, we start by estimating J from sample averages and derive the objective ˆ J as follows:
[Equation]
H
[Equation]
It can easily be seen that to maximize the above expression E (cid:2) T t | H t = h t , π, O (cid:3) should be minimized while Pr( H = h | π, O ) should be maximized.
Given that for long trajectories the expected number of terminations increases while the probability of generating the trajectories goes to 0 , we normalize the number of terminations by the lenght of the trajectory, | h | , and adjust a hyperparameter, λ 2 , to prevent one term from dominating the other during optimization.
Based on this observation we propose optimizing the following objective:
[Equation]
This objective allow us to control a trade-off, through λ 2 , of how much we care about the options reproducing the demonstrated trajectories vs. how much we want the agent to minimize the number of decisions.
A. A PPENDIX B - P ROOF OF T HEOREM
Theorem 1 Given a set of options O and a policy π over options, the expected number of terminations for a trajectory h of length | h | is given by:
[Equation]
where ,
Pr
[Equation]
and Pr( O
= o | H
= h 0 , π, O )
=
π ( s 0 , o ) .
Proof.
Notice that (cid:80) | h | t =1 E [ T t = 1 | H t
=
h t , π, O ] = (cid:80)
| h | t =1 Pr( T t = 1 | H t = h t , π, O ) 1 , so if we ﬁnd an expression for Pr( T t = 1 | H t
= h t , π, O ) , we can calculate the expectation exactly.
We deﬁne Pr( T 0 = 1 | H 1 =
h 1 , π, O ) = 1 for ease of derivation even though there is no option to terminate at T 0 .
Pr(
[Equation]
O
[Equation]
o ∈O
Pr(
[Equation]
O t
[Equation]
t
[Equation]
t
[Equation]
O )
[Equation] [Equation]
t =
[Equation]
t ,
[Equation]
O )
= (cid:88)
o ∈O β
o
[Equation] [Equation]
O
[Equation]
−
[Equation]
O )
= (cid:88)
o ∈O β
o
[Equation] [Equation]
O t
[Equation] [Equation]
t − 1 = h t −
[Equation]
A t − 1 = a t −
[Equation]
S t = s t ,
[Equation]
O )
= (cid:88)
o ∈O β
o
( s t )
Pr( S t =
s t | H t − 1 =
h t − 1 , A t − 1 = a t − 1 , O t −
= o, π, O )
Pr( S t = s t | H t − 1 =
h t − 1 , A t − 1 = a t − 1 , π, O )
× Pr( O t − 1 =
o | H t − 1 =
h t − 1 , A t − 1 = a t − 1 , π, O ) = (cid:88) o ∈O β
o
( s t )
[Equation]
( S t
[Equation]
t
[Equation]
t −
[Equation]
t − 1 , A t −
[Equation]
a t − 1 ,
[Equation]
, O )
Pr( S t = s t |
H t −
[Equation]
h t − 1 , A t −
[Equation]
a t − 1 , π, O ) ×
[Equation]
( O t − 1 =
o | H t − 1 =
h t − 1 , A t − 1 = a t − 1 , π, O ) = (cid:88) o ∈O β
o
( s t ) Pr( O t − 1 =
o | H t − 1 =
h t − 1 , A t − 1 = a t − 1 , π, O ) = (cid:88)
[Equation]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We are left with ﬁnding an expression in terms of known probabilities for Pr( O t −
= o | H t − 1 =
h t − 1 , π, O ) .
[Equation]
Given that by convention, Pr( T 0 = 1 | H
= h 0 , π, O )
= 1 . 0 , we are now left with ﬁguring out how to calculate Pr( O t −
=
o | H t − 1 =
h t − 1 , T t − 1 = 0 , π, O )
Pr( O t − 1 =
o | H t − 1 =
h t − 1 , T t − 1 = 0 , π, O ) =
Pr( O t − 2 =
o, A t − 2 =
a t − 2 , S t − 1 =
s t −
[Equation]
π
[Equation]
O
[Equation]
where Pr( O 0 = o | H
= h 0 , π, O )
=
π ( s 0 , o )
Using the recursive function Pr( O t −
= o (cid:48) | H t − 1 =
h t − 1 , π, O ) , the expected number of terminations for a given trajectory is given by:
[Equation]
A. A PPENDIX C - P ROOF OF T HEOREM
Theorem
Given a set of options O and a policy π over options, the probability of generating a trajectory h of length | h | is given by:
[Equation]
where f is a recursive function deﬁned as:
[Equation]
µ o ( s i +1 , a i +1 ) f
( h t , o, i + 1) (cid:21) , otherwise
Proof.
We deﬁne H i,t to be the history from time i to time t , that is, H i,t =
( S i , A i , S
i +1 ,
A i +1 , . . .
, S t ), where i < t .
If i = t , the history would contain a single state.
[Equation]
o ∈O
[Equation]
o ∈O
[Equation]
o ∈O
[Equation]
o ∈O
[Equation]
, O
= o,
A 0 = a 0 , π, O ) .
We now need to ﬁnd an expression to calculate Pr( H 1 ,t =
h 1 ,t | S
= s 0 , O
= o,
A 0 = a 0 , π, O ) .
Consider the probability of seeing history h
i,t given the previous state, s , the previous option, o , and the previous action, a :
Pr( H i,t = h
i,t | S i − 1 = s, O i −
= o,
A i − 1 = a ) =
Pr( S i = s i | S
i −
= s, O i −
= o,
A i − 1 = a )
Pr( H i +1 ,t
= h
i +1 ,t , A
i = a i | S i −
= s, O i −
= o,
A i − 1 = a, S i =
s i )
=
P ( s, a, s i )
Pr( H i +1 ,t = h
i +1 ,t , A
i = a i | S i −
= s, O i −
= o,
A i − 1 = a, S i =
s i )
=
P ( s, a, s i )
Pr( H i +1 ,t = h
i +1 ,t , A
i = a i
| O i −
= o,
A i − 1 = a, S i =
s i )
=
P ( s, a, s i ) (cid:2)
Pr( T i =
|
O i −
= o,
A i − 1 = a, S i =
s i ) ×
Pr( H i +1 ,t = h
i +1 ,t , A
i = a i
| O i −
= o,
A i − 1 = a, S i =
s i , T i = 1)
+ Pr( T i =
| O i −
= o,
A i − 1 = a, S i =
s i ) ×
Pr( H i +1 ,t = h
i +1 ,t , A
i = a i
| O i −
= o,
A i − 1 = a, S i =
s i , T i = 0) (cid:3) =
P ( s, a, s i ) (cid:2) β
o ( s i ) ×
Pr( H i +1 ,t = h
i +1 ,t , A
i = a i
| O i −
= o,
A i − 1 = a, S i =
s i , T i = 1)
+ (1 − β o
( s i ))
×
Pr( H i +1 ,t = h i +
[Equation]
Even though the equation above might seem complicated, there are only two cases we need to consider: either the current option terminates and a new one must be selected (the ﬁrst term), or the current option does not terminate (the second term).
Let’s consider each of them separately.
Case 1 - option terminates: If we terminate, we sum over new options:
[Equation]
∈O
[Equation]
1) × Pr( H i +
[Equation]
t = h i +
[Equation]
t
[Equation]
A i =
[Equation]
i
[Equation]
O i −
[Equation]
o
[Equation]
A i −
[Equation]
a
[Equation]
S i =
s i
[Equation]
T i = 1 , O i
= o (cid:48) )
= (cid:88) o (cid:48)
∈O π ( s i , o (cid:48) )
Pr( H i
[Equation] [Equation]
t
[Equation]
h
i
[Equation] [Equation]
t , A i
[Equation]
a i |
O i −
= o,
A i − 1 = a, S i =
s i , T i = 1 , O i
= o (cid:48) )
= (cid:88) o (cid:48)
∈O π ( s i , o (cid:48) )
[Equation]
( H i
[Equation] [Equation]
t = h
i
[Equation] [Equation]
t
[Equation]
A i = a i | S i
[Equation]
s i , O i
= o (cid:48) )
= (cid:88) o (cid:48)
∈O π (
[Equation]
(cid:48) )
Pr( A
[Equation]
a
[Equation]
S
[Equation]
s
[Equation]
O
[Equation]
o (cid:48) )
Pr( H i
[Equation] [Equation]
t = h
i +1 ,t | S i
= s i , O
[Equation]
Note that the expanded probability has the same form as Pr( H i,t
= h i,t | S i − 1 = s, O i −
= o,
A i − 1 = a ) .
Case 2 - option does not terminate: This tells us that O i
=
o , so we may drop the dependency on the i − 1 terms:
[Equation]
Plugging these two cases back into our earlier equation yields:
[Equation]
Note that each term contains an expression of the same form, Pr( H i,t = h
i,t | S i − 1 = s, O i −
= o,
A i − 1 = a ) .
We can therefore compute the probability recursively.
Our recursion will terminate when we consider i
=
t , as H t,t contains a single state, and we adopt the convention of its probability to be 1.
Notice that for every recursive step, both inner terms will produce a P ( s, a, s i ) term.
Consider the result when we factor every recursive P ( s, a, s i ) term to the front of the equation.
We deﬁne the following recursive function:
[Equation]
otherwise .
Notice that this is the recursive probability described above, but with the P ( s, a, s (cid:48) ) terms factored out.
We now see that:
[Equation]
Plugging this all the back into our original equation for Pr( H t = h t | π, O ) gives us the desired result:
[Equation]
A. A PPENDIX D - E MPIRICAL V ALIDATION OF D ERIVED E QUATIONS
To double check the derivation of the proposed objective and make sure the implementation was correct, we conducted a simple empirical test to compared the calculated expected number of decisions in a trajectory and the probability of generating each trajectory for a set of 10 trajectories on 10 MDPs.
The MDPs are simple chains of 7 states with different transition functions.
We randomly initialized four options and a policy over options, and estimated the probability of generating each trajectory and the expected number of terminations, for each sampled trajectory, by Montecarlo sampling 10 , 000 trials.
Table 1 presents results for the 10 trajectories verifying empirically that the equations were correctly derived and implemented.
The table compares the empirical and true probability of generating a given trajectory, ˆPr( H |· ) and
Pr( H |· ) , respectively, and the empirical and true sum of expected number of decisions an agent has to make to generate those trajectories, (cid:80)
| H | t =1 ˆ E
[ T t |· ] and (cid:80)
[Table]
Note that the cases with largest discrepancy between the estimated and calculated number of terminations occur when the probability of generating a trajectory is low.
This happens because, since the trajectory is unlikely to be generated, the Monte Carlo sampling is not able to produce enough samples of the trajectory.
A. A PPENDIX E - I MPLEMENTATION D ETAILS FOR A TARI E XPERIMENTS
For these experiments we ﬁrst learned a good performing policy with A3C for each game and sampled 12 trajectories for training.
Each trajectory lasted until a life was lost, not for the entire duration of the episode.
Each option was represented as a two-layer neural network, with 32 neurons in each layer
[Table]
and two output layers: a softmax output layer over the four possible actions representing µ , and a separate sigmoid layer representing β .
We implemented our objective using PyTorch which simpliﬁes gradient calculations.
The options were represented by a two-layer neural network, where the input was represented by gray scale images of the last two frames.
We ran 32 training agents in parallel on CPUs, the learning rate was set to 0 . 0001 and the discount factor γ was set to 0 .
Because the options can only learn from the states observed in the trajectories, it is possible that when using them, they will be executed in previously unseen states.
When this happens, the termination function may decide to never terminate, as it has not seen that region of the state space before.
To address this issue, we add a value of 0 .
05 to the predicted probability of termination per time-step that the option has been running since executed.
Therefore, in our experiments an option cannot run for more than 20 time-steps in total.