A UTOMATA G UIDED H IERARCHICAL R EINFORCEMENT L EARNING FOR Z ERO -S HOT S KILL C OMPOSITION
Anonymous authors
Paper under double-blind review
A BSTRACT
An obstacle that prevents the wide adoption of (deep) reinforcement learning (RL) in control systems is its need for a large number of interactions with the environment in order to master a skill.
The learned skill usually generalizes poorly across domains and re-training is often necessary when presented with a new task.
We present a framework that combines techniques in formal methods with hierarchical reinforcement learning (HRL).
The set of techniques we provide allows for the convenient speciﬁcation of tasks with logical expressions, learns hierarchical policies (meta-controller and low-level controllers) with well-deﬁned intrinsic rewards using any RL methods and is able to construct new skills from existing ones without additional learning.
We evaluate the proposed methods in a simple grid world simulation as well as simulation on a Baxter robot.
I NTRODUCTION
Reinforcement learning has received much attention in the recent years because of its achievements in games Mnih et al.
(2015), Silver et al.
robotics manipulation Jang et al., Levine et al.
(2016), Gu et al.
and autonomous driving Isele et al.
(2017), Madrigal (2017).
However, training a policy that sufﬁciently masters a skill requires an enormous amount of interactions with the environment and acquiring such experience can be difﬁcult on physical systems.
Moreover, most learned policies are tailored to mastering one skill (by maximizing the reward) and are hardly reusable on a new skill.
Skill composition is the idea of constructing new skills out of existing skills (and hence their policies) with little to no additional learning.
In stochastic optimal control, this idea has been adopted by authors of Todorov (2009) and Da Silva et al.
(2009) to construct provably optimal control laws based on linearly solvable Markov decision processes.
Authors of Haarnoja et al.
(2017), Tang & Haarnoja have showed in simulated manipulation tasks that approximately optimal policies can result from adding the Q-functions of the existing policies.
Hierarchical reinforcement learning is an effective means of achieving transfer among tasks.
The goal is to obtain task-invariant low-level policies, and by re-training the meta-policy that schedules over the low-level policies, different skills can be obtain with less samples than training from scratch.
Authors of Heess et al.
(2016) have adopted this idea in learning locomotor controllers and have shown successful transfer among simulated locomotion tasks.
Authors of Oh et al.
(2017) have utilized a deep hierarchical architecture for multi-task learning using natural language instructions.
Temporal logic is a formal language commonly used in software and digital circuit veriﬁcation Baier & Katoen (2008) as well as formal synthesis Belta et al.
It allows for convenient expression of complex behaviors and causal relationships.
TL has been used by Sadraddini & Belta (2015),
Leahy et al.
(2015) to synthesize provably correct control policies.
Authors of Aksaray et al.
(2016) have also combined TL with Q-learning to learn satisﬁable policies in discrete state and action spaces.
In this work, we focus on hierarchical skill acquisition and zero-shot skill composition.
Once a set of skills is acquired, we provide a technique that can synthesize new skills without the need to further interact with the environment (given the state and action spaces as well as the transition remain the same).
We adopt temporal logic as the task speciﬁcation language.
Compared to most heuristic reward structures used in the RL literature to specify tasks, formal speciﬁcation language excels at its semantic rigor and interpretability of speciﬁed behaviors.
Our main contributions are:
• We take advantage of the transformation between TL formula and ﬁnite state automata (FSA) to construct deterministic meta-controllers directly from the task speciﬁcation without the necessity for additional learning.
We show that by adding one discrete dimension to the original state space, structurally simple parameterized policies such as feed-forward neural networks can be used to learn tasks that require complex temporal reasoning.
• Intrinsic motivation has been shown to help RL agents learn complicated behaviors with less interactions with the environment Singh et al.
(2004), Kulkarni et al.
(2016), Jaderberg et al.
However, designing a well-behaved intrinsic reward that aligns with the extrinsic reward takes effort and experience.
In our work, we construct intrinsic rewards directly from the input alphabets of the FSA (a component of the automaton), which guarantees that maximizing each intrinsic reward makes positive progress towards satisfying the entire task speciﬁcation.
From a user’s perspective, the intrinsic rewards are constructed automatically from the TL formula.
• In our framework, each FSA represents a hierarchical policy with low-level controllers that can be re-modulated to achieve different tasks.
Skill composition is achieved by manipulating the FSA that results from their TL speciﬁcations in a deterministic fashion.
Instead of interpolating/extrapolating among existing skills, we present a simple policy switching scheme based on graph manipulation of the FSA.
Therefore, the compositional outcome is much more transparent.
We introduce a method that allows learning of such hierarchical policies with any non-hierarchical RL algorithm.
Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class.
P RELIMINARIES 2. T HE O PTIONS F RAMEWORK IN H IERARCHICAL R EINFORCEMENT LEARNING
In this section, we brieﬂy introduce the options framework Sutton et al.
(1998), especially the terminologies that we will inherit in later sections.
We start with the deﬁnition of a Markov Decision
Process.
Deﬁnition 1.
An MDP is deﬁned as a tuple M = (cid:104) S, A, p ( ·|· , · ) , R ( · , · , · ) (cid:105) , where S ⊆
IR n is the state space ; A ⊆ IR m is the action space ( S and A can also be discrete sets); p : S ×
A × S → [0 , 1] is the transition function with p ( s (cid:48) | s, a ) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s (cid:48) ∈ S ; R : S ×
A × S → IR is the reward function.
let T be the length of a ﬁxed time horizon.
The goal is to ﬁnd a policy π (cid:63) : S → A (or π (cid:63) : S
×
A → [0 , 1] for stochastic policies) that maximizes the expected return, i.e.
[Equation]
where τ T =
( s 0 , a 0 , ..., s T , ) denotes the state-action trajectory from time 0 to T .
The options framework exploits temporal abstractions over the action space.
An option is deﬁned as a tuple o =
(cid:104)I , π o , β (cid:105) where I is the set of states that option o can be initiated (here we let
I = S for all options), π o : S
→ A is an options policy and β : S → [0 , 1] is the termination probability for the option at state s .
In addition, there is a policy over options π h : S → O (where
O is a set of available options) that schedules among options.
At a given time step t , an option o is chosen according to π h ( s t ) and the options policy π o is followed until the termination probability β ( s ) > threshold at time t + k , and the next option
2. SC TLTL A ND A UTOMATA
We consider tasks speciﬁed with Truncated Linear Temporal Logic (TLTL).
We restrict the set of allowed operators to be
[Equation]
where f ( s ) < c is a predicate, ¬ (negation/not), ∧ (conjunction/and), and ∨ (disjunction/or) are
Boolean connectives, and ♦ (eventually), U (until), T (then), (cid:13) (next), are temporal operators.
Implication is denoted by ⇒ (implication).
Essentially we excluded the Always operator ( (cid:50) ) with reasons similar to Kupferman & Vardi (2001).
We refer to this restricted TLTL as syntactically co-safe TLTL (scTLTL) (Vasile et al.
(2017) used similar idea for LTL).
There exists a real-value function ρ ( s 0: T , φ ) called robustness degree that measures the level of satisfaction of trajectory s 0: T with respective to φ .
ρ ( s 0: T , φ ) > 0 indicates that s 0: T satisﬁes φ and vice versa.
Deﬁnitions for the boolean semantics and robustness degree are provided in Appendix E.
Any scTLTL formula can be translated into a ﬁnite state automata (FSA) with the following deﬁnition:
Deﬁnition 2.
An FSA is deﬁned as a tuple A φ = (cid:104) Q φ , Ψ φ , q 0 , p φ ( ·|· ) , F φ (cid:105) , where Q φ is a set of automaton states; Ψ φ is an input alphabet, we denote ψ q i ,q j ∈ Ψ
φ the predicate guarding the transition from q i to q j (as illustrated in Figure 1 ); q 0 ∈ Q φ is the initial state; p φ : Q φ × Q φ → [0 , 1] is a conditional probability deﬁned as
[Equation]
In addition, given an MDP state s , we can calculate the transition in automata states at s by
[Equation]
We abuse the notation p φ to represent both kinds of transitions when the context is clear.
F φ is a set of ﬁnal automaton states.
The translation from TLTL formula to FSA to can be done automatically with available packages like Lomap Ulusoy (2017).
Example 1.
Figure 1 (left) illustrates the FSA resulting from formula φ = ¬ b U a .
In English, φ entails during a run, b cannot be true until a is true and a needs to be true at least once.
The FSA has three automaton states Q φ = { q 0 , q f , trap } with q 0 being the input(initial) state (here q i serves to track the progress in satisfying φ ).
The input alphabet is deﬁned as the Ψ φ = {¬ a ∧ ¬ b, ¬ a ∧ b, a ∧¬ b, a ∧ b } .
Shorthands are used in the ﬁgure, for example a = ( a ∧ b ) ∨ ( a ∧¬ b ) .
Ψ φ represents the power set of { a, b } , i.e. Ψ φ = 2 { a,b } .
During execution, the FSA always starts from state q 0 and transitions according to Equation (3) or (4) .
The speciﬁcation is satisﬁed when q f is reached and violated when trap is reached.
In this example, q f is reached only when a becomes true before b becomes true.
P ROBLEM F ORMULATION A ND A PPROACH
We start with the following problem deﬁnition:
Problem 1.
Given an MDP in Deﬁnition 1 with unknown transition dynamics p ( s (cid:48) | s, a ) and a scTLTL speciﬁcation φ over state predicates (along with its FSA A φ ) as in Deﬁnition 2.
Find a policy π (cid:63)φ such that
[Equation]
where 1 ( ρ ( s 0: T , φ ) > 0) is an indicator function with value 1 if ρ ( s 0: T , φ ) > 0 and 0 otherwise.
Problem 1 deﬁnes a policy search problem where the trajectories resulting from following the optimal policy should satisfy the given scTLTL formula in expectation.
Problem 2.
Given two scTLTL formula φ 1 and φ 2 along with policy π φ 1 that satisﬁes φ 1 and π φthat satisﬁes φ 2 .
Obtain a policy π φ that satisﬁes φ = φ 1 ∧ φ 2 .
Problem 2 deﬁnes the problem of task composition.
Given two policies each satisfying a scTLTL speciﬁcation, construct the policy that satisﬁes the conjunction of the given speciﬁcations.
Solving this problem is useful when we want to break a complex task into simple and manageable components, learn a policy that satisﬁes each component and ”stitch” all the components together so that the original task is satisﬁed.
It can also be the case that as the scope of the task grows with time, the original task speciﬁcation is amended with new items.
Instead of having to re-learn the task from scratch, we can only learn a policy that satisﬁes the new items and combine them with the old policy.
We propose to solve Problem 1 by constructing a product MDP from the given MDP and FSA that can be solved using any state-of-the-art RL algorithm.
The idea of using product automaton for control synthesis has been adopted in various literature Leahy et al.
(2015), Chen et al.
However, the methods proposed in these works are restricted to discrete state and actions spaces.
We extend this idea to continuous state-action spaces and show its applicability on robotics systems.
For Problem 2, we propose a policy switching scheme that satisﬁes the compositional task speciﬁcation.
The switching policy takes advantage of the characteristics of FSA and uses robustness comparison at each step for decision making.
FSA A UGMENTED MDP
Problem 1 can be solved with any episode-based RL algorithm.
However, doing so the agent suffers from sparse feedback because a reward signal can only be obtained at the end of each episode.
To address this problem as well as setting up ground for automata guided HRL, we introduce the FSA augmented MDP
Deﬁnition 3.
An FSA augmented MDP corresponding to scTLTL formula φ is deﬁned as M φ = (cid:104)
˜ S, A, ˜ p ( ·|· , · ) , ˜ R ( · , · ) (cid:105) where ˜ S
⊆ S × Q φ , A is the same as the original MDP.
˜ p
(˜ s (cid:48) | ˜ s, a ) is the probability of transitioning to ˜ s (cid:48) given ˜ s and a , in particular
[Equation]
Here p φ is deﬁned in Equation (4) .
˜
R : ˜ S × ˜ S → IR is the FSA augmented reward function, deﬁned by
[Equation] [Equation]
where Ω q is the set of automata states that are connected with q through outgoing edges.
D qφ = (cid:87) q (cid:48) ∈
Ω q ψ q,q (cid:48) represents the disjunction of all predicates guarding the transitions that originate from q .
The goal is to ﬁnd the optimal policy that maximizes the expected sum of discounted return, i.e.
[Equation]
where γ < 1 is the discount factor, T is the time horizon.
As a quick example to the notation D qφ , consider the state q 0 in the FSA in Figure 1 , Ω q 0 = { trap, q f } , D q 0 φ = ψ q 0 ,trap ∨ ψ q 0 ,qf = b ∨ a .
The goal is then to ﬁnd a policy π : ˜ S
→ A that maximizes the expected sum of ˜ R over the horizon T .
The FSA augmented MDP can be constructed with any standard MDP and a scTLTL formula.
And it can be solved with any off-the-shelf RL algorithm.
By directly learning the ﬂat policy π we bypass the need to learn multiple options policies separately.
After obtaining the optimal policy π (cid:63) , the optimal options policy for any option o q can be extracted by executing π (cid:63) ( a | s, q ) without transitioning the automata state, i.e. keeping q i ﬁxed (denoted π (cid:63)q ).
And π (cid:63)q satisﬁes
[Equation] [Equation]
In other words, the purpose of π q i is to activate one of the outgoing edges of q i as soon as possible and by doing so repeatedly eventually reach q f .
The reward function in Equation (7) encourages the system to exit the current automata state and move on to the next, and by doing so eventually reach the ﬁnal state q f .
However, this reward does not distinguish between the trap state and other states and therefore will also promote entering of the trap state.
One way to address this issue is to impose a terminal reward on both q f and trap .
Because the reward is an indicator function with maximum value of 1, we assign terminal rewards
R q f = 2 and R trap = − 2 .
Appendix
D describes the typical learning routine using FSA augmented MDP.
The algorithm utilizes memory replay which is popular among off-policy RL methods (DQN, A3C, etc) but this is not a requirement for learning with ˜ M φ .
On-policy methods can also be used.
A UTOMATA G UIDED T ASK C OMPOSITION
In section, we provide a solution for Problem 2 by constructing the FSA of φ from that of φ 1 and φand using φ to synthesize the policy for the combined skill.
We start with the following deﬁnition.
Deﬁnition 4.
Given A φ 1 = (cid:104) Q
[Equation]
1 , Ψ
[Equation]
1 , q 01 , p φ 1 , F φ 1 (cid:105) and A φ 2 = (cid:104) Q φ 2 , Ψ φ 2 , q 02 , p φ 2 , F φ 2 (cid:105) , The FSA of φ is the product automaton of A φ 1 and A φ 1 , i.e. A φ = φ 1 ∧ φ 2 =
A φ 1 × A φ 2 = (cid:104) Q φ , Ψ φ , q 0 , p φ , F φ (cid:105) where Q φ ⊆ Q φ 1 × Q φ 2 is the set of product automaton, states, q 0 = ( q 01 , q 02 ) is the product initial state, F ⊆ F φ 1 ∩ F φ 2 is the ﬁnal accepting states.
Following Deﬁnition 2, for states q = ( q 1 , q 2 ) ∈ Q φ and q (cid:48) =
( q (cid:48) 1 , q (cid:48)
) ∈ Q φ , the transition probability p φ is deﬁned as
[Equation]
Example 2.
Figure 1 (right) illustrates the FSA of A φ 1 and A φ 2 and their product automaton A φ .
Here φ 1 = ♦ a ∧ ♦ b which entails that both a and b needs to be true at least once (order does not matter), and φ 2 = ¬
b U a which is the same as Example 1.
The resultant product corresponds to the formula φ = ( ♦ a ∧♦ b ) ∧ ( ¬ b U a ) which dictates that a and b need to be true at least once, and a needs to be true before b becomes true (an ordered visit).
We can see that the trap state occurs in A φ 2 and A φ , this is because if b is ever true before a is true, the speciﬁcation is violated and q f can never be reached.
In the product automaton, we aggregate all state pairs with a trap state component into one trap state.
For q = ( q 1 , q 2 ) ∈ Q φ , let Ψ q , Ψ q 1 and Ψ q 2 denote the set of predicates guarding the outgoing edges of q , q 1 and q 2 respectively.
Equation (10) entails that a transition at q in the product automaton
A φ exists only if corresponding transitions at q 1 , q 2 exist in A φ 1 and A φ 2 respectively.
Therefore, ψ q,q (cid:48) =
ψ q 1 ,q (cid:48) 1 ∧ ψ q 2 ,q (cid:48) 2 , for ψ q,q (cid:48)
∈
Ψ q , ψ q 1 ,q (cid:48) 1 ∈ Ψ q 1 , ψ q 2 ,q (cid:48) 2 ∈ Ψ q 2 (here q (cid:48)
i is a state such that p φ i ( q (cid:48)
i | q i )
= 1 ).
Following Equation (9),
[Equation]
where D qφ = (cid:95)
[Equation]
Repeatedly applying the distributive law (∆ ∧ Ω 1 ) ∨ (∆ ∧ Ω 2 ) =
∆ ∧ (Ω 1 ∨ Ω 2 ) to the logic formula
D qφ transforms the formula to
[Equation]
Therefore,
[Equation]
The second step in Equation (13) follows the robustness deﬁnition.
Recall that the optimal options policies for q 1 and q 2 satisfy
[Equation]
Equation (13) provides a relationship among π (cid:63)q , π (cid:63)q 1 and π (cid:63)q 2 .
Given this relationship
[Equation]
We propose a simple switching policy based on stepwise robustness comparison that satisﬁes φ = φ 1 ∧ φ 2 as follows
[Equation]
We show empirically the use of this switching policy for skill composition and discuss its limitations in the following sections.
E XPERIMENTS A ND D ISCUSSION 6. G RID W ORLD S IMULATION
In this section, we provide a simple grid world navigation example to illustrate the techniques presented in Sections 4 and 5.
Here we have a robot navigating in a discrete 1 dimensional space.
Its
MDP state space S =
{ s | s ∈ [ − 5 , 5) , s is discrete } ,
its action space A
= { left, stay, right } .
The robot navigates in the commanded direction with probability 0.8, and with probability 0.2 it randomly chooses to go in the opposite direction or stay in the same place.
The robot stays in the same place if the action leads it to go out of bounds.
We deﬁne two regions a : − 3 < s < − 1 and b : 2 < s < 4 .
For the ﬁrst task, the scTLTL speciﬁcation φ 1 = ♦ a ∧ ♦ b needs to be satisﬁed.
In English, φ 1 entails that the robot needs to visit regions a and b at least once.
To learn a deterministic optimal policy π (cid:63)φ 1 : S × Q → A , we use standard Q-Learning Watkins (1989) on the FSA augmented MDP for this problem.
We used a learning rate of 0 . 1 , a discount factor of 0.99, epsilon-greedy exploration strategy with (cid:15) decaying linearly from 0.0 to 0.01 in 1500 steps.
The episode horizon is T = 50 and trained for 500 iterations.
All Q-values are initialized to zero.
The resultant optimal policy is illustrated in Figure 2 .
We can observe from the ﬁgure above that the policy on each automaton state q serves a speciﬁc purpose.
π (cid:63)q 0 tries to reach region a or b depending on which is closer.
π (cid:63)q 1 always proceeds to region a .
π (cid:63)q 2 always proceeds to region b .
This agrees with the deﬁnition in Equation 9.
The robot can start anywhere on the s axis but must always start at automata state q 0 .
Following π φ 1 , the robot will ﬁrst reach region a or b (whichever is nearer), and then aim for the other region which in turn satisﬁes φ .
The states that have stay as their action are either goal regions (states ( − 2 , q 0 ) , (3 , q 1 ) , etc) where a transition on q happens or states that are never reached (states ( − 3 , q 1 ) , ( − 4 , q 2 ) , etc) because a transition on q occurs before they can be reached.
To illustrate automata guided task composition described in Example 2, instead of learning the task described by φ from scratch, we can simply learn policy π φ 2 for the added requirement φ 2 =
¬
b U a .
We use the same learning setup and the resultant optimal policy is depicted in Figure.
It can be observed that π φ 2 tries to reach a while avoiding b .
This behavior agrees with the speciﬁcation φ 2 and its FSA provided in Figure 2 .
The action at s = 4 is stay because in order for the robot to reach a it has to pass through b , therefore it prefers to obtain a low reward over violating the task.
Having learned policies π φ 1 and π φ 2 , we can now use Equation 15 to construct policy π φ 1 ∧ φ 2 .
The resulting policy for π φ 1 ∧ φ 2 is illustrated in Figure 2 (upper right).
This policy guides the robot to ﬁrst reach a (except for state s = 4 ) and then go to b which agrees with the speciﬁcation.
Looking at Figure 1 , the FSA of φ = φ 1 ∧ φ 2 have two options policies π φ ( · , q 0 ) and π φ ( · , q 1 ) 2 ( trap state and q f are terminal states which don’t have options).
State q 1 has only one outgoing edge with the guarding predicate ψ q 1 ,q f : b , which means π φ ( · , q 1 ) =
π φ 1 ( · , q 2 ) (they have the same guarding predicate).
Policy π φ ( · , q 0 ) is a switching policy between π φ 1 ( · , q 0 ) and π φ 2 ( · , q 0 ) .
Figure 2 (lower left) shows the robustness comparison at each state.
The policy with lower robustness is chosen following Equation (15).
We can see that the robustness of both policies are the same from s = −to s = 0 .
And their policies agree in this range (Figures 3 and 4 ).
As s becomes larger, disagreement emerge because π φ 1 ( · , q 0 ) wants to stay closer to b but π φ 2 ( · , q 0 ) wants otherwise.
To maximize the robustness of their conjunction, the decisions of π φ 2 ( · , q 0 ) are chosen for states s > 0 .
6. S IMULATED B AXTER
In this section, we construct a set of more complicated tasks that require temporal reasoning and evaluate the proposed techniques on a simulated Baxter robot.
The environment is shown in Figure( left ).
In front of the robot are three square regions and two circular regions.
An object with planar coordinates p = ( x, y ) can use predicates S red ( p ) , S blue ( p ) , S black ( p ) , C red ( p ) , C blue ( p ) to evaluate whether or not it is within the each region.
The predicates are deﬁned by S : ( x min < x < x max ) ∧ ( y min <
y < y max ) and C : dist (( x, y ) , ( x, y ) center ) <
r .
( x min , y min ) and ( x max , y max ) are the boundary coordinates of the square region, ( x, y ) center and r are the center and radius of the circular region.
There are also two boxes which planar positions are denoted as p redbox =
( x, y ) redbox and p bluebox =
( x, y ) bluebox .
And lastly there is an interactive ball that a user can move in space which 2D coordinate is denoted as p sphere =
( x, y ) sphere (all objects move in the table plane).
We design seven tasks each speciﬁed by a scTLTL formula.
The task speciﬁcations and their English translations are provided in Appendix A. Throughout the experiments in this section, we use proximal policy search Schulman et al.
(2017) as the policy optimization method.
The hyperparameters are kept ﬁxed across the experiments and are listed in Appendix B.
The policy is a Gaussian distribution parameterized by a feed-forward neural network with 2 hidden layers, each layer hasrelu units.
The state and action spaces vary across tasks and comparison cases, and are described in
Appendix C.
We use the ﬁrst task φ 1 to evaluate the learning outcome using the FSA augmented MDP.
As comparisons, we design two other rewards structures.
The ﬁrst is to use the robustness ρ ( s 0: T , φ ) as the terminal reward for each episode and zero everywhere else, the second is a heuristic reward that aims to align with φ 1 .
The heuristic reward consists of a state that keeps track of whether the sphere is in a region and a set of quadratic distance functions.
For φ 1 , the heuristic reward is
[Equation]
Heuristic rewards for other tasks are deﬁned in a similar manner and are not presented explicitly.
The results are illustrated in Figure 3 ( right ).
The upper right plot shows the average robustness over training iterations.
Robustness is chosen as the comparison metric for its semantic rigor (robustness greater than zero satisﬁes the task speciﬁcation).
The reported values are averaged over 60 episodes and the plot shows the mean and 2 standard deviations over 5 random seeds.
From the plot we can observe that the FSA augmented MDP and the terminal robustness reward performed comparatively in terms of convergence rate, whereas the heuristic reward fails to learn the task.
The FSA augmented
MDP also learns a policy with lower variance in ﬁnal performance.
We deploy the learned policy on the robot in simulation and record the task success rate.
For each of the three cases, we deploy the 5 policies learned from 5 random seeds on the robot and perform 10 sets of tests with randomly initialized states resulting in 50 test trials for each case.
The average success rate is presented in Figure 3 ( lower right ).
From the results we can see that the FSA augmented MDP is able to achieve the highest rate of success and this advantage over the robustness reward is due to the low variance of its ﬁnal policy.
To evaluate the policy switching technique for skill composition, we ﬁrst learn four relatively simple policies π φ 2 , π φ 3 , π φ 4 , π φ 5 using the FSA augmented MDP.
Then we construct π φ 6 =
π
φ 2 ∧ φ 3 and π φ 7 = π φ 2 ∧ φ 3 ∧ φ 4 ∧ φ 4 using Equation (15) (It is worth mentioning that the policies learned by the robustness and heuristic rewards do not have an automaton state in them, therefore the skill composition technique does not apply).
We deploy π φ 6 and π φ 7 on tasks 6 and 7 for 10 trials and record the average robustness of the resulting trajectories.
As comparisons, we also learn tasks 6 and 7 from scratch using terminal robustness rewards and heuristic rewards, the results are presented in Figure.
We can observe from the plots that as the complexity of the tasks increase, using the robustness and heuristic rewards fail to learn a policy that satisﬁes the speciﬁcations while the constructed policy can reliably achieve a robustness of greater than zero.
We perform the same deployment test as previously described and looking at Figure 4 ( right ) we can see that for both tasks 6 and 7, only the policies constructed by skill composition are able to consistently complete the tasks.
C ONCLUSION
In this paper, we proposed the FSA augmented MDP, a product MDP that enables effective learning of hierarchical policies using any RL algorithm for tasks speciﬁed by scTLTL.
We also introduced automata guided skill composition, a technique that combines existing skills to create new skills without additional learning.
We show in robotic simulations that using the proposed methods we enable simple policies to perform logically complex tasks.
Limitations of the current framework include discontinuity at the point of switching (for Equation (15)), which makes this method suitable for high level decision tasks but not for low level control tasks.
The technique only compares robustness at the current step and chooses to follow a sub-policy for one time-step, making the switching policy short-sighted and may miss long term opportunities.
One way to address this is to impose a termination condition for following each sub- policy and terminate only when the condition is triggered (as in the original options framework).
This termination condition can be hand designed or learned
B H YPERPARAMETERS F OR P ROXIMAL P OLICY O PTIMIZATION
[Table]
[Table]
C S TATE A ND A CTION S PACES
For experiments with the simulated Baxter robot, we delegate low level control to motion planning packages and only learn high level decisions.
Depending on the task, the states are the planar positions of objects (red box, blue box, ball) and the automata state.
The actions are the target positions of the objects.
We assume that the low level controller can take objects to the desired target position with minor uncertainty that will be dealt with by the learning agent.
The table below shows state and action spaces used for each task.
s M and a M denote the spaces for regular MDP (used for terminal robustness rewards and heuristic rewards).
s ˜ M and a ˜ M denote the spaces for
FSA augmented MDP.
q denotes the automata state.
[Table]
State Space
Action Space
[Equation]
p redbox ) target
[Equation]
a
[Equation]
target
[Equation]
a
[Equation]
target
[Equation]
target
[Equation]
a
[Equation]
a
[Equation]
For tasks φ 6 and φ 7 , the action space is three dimensional, the ﬁrst two dimension p =
( x, y ) is a target position, the third dimension d controls which object should be placed at p .
If d < 0 .
5 , then p = p redbox and if d > 0 .
5 , then p = p bluebox .
D L EARNING W ITH FSA A UGMENTED MDP ( OFF - POLICY VERSION )
parameterized policy
3: Initialize replay pool B ← {} 4: for n = 1 to number of training episodes do 5:
Select initial state ˜ s
= ( s 0 , q 0 ) (cid:46) s 0 can be randomly selected, q 0 is always the initial automaton state q6: for t =0 to T do a t
= π
(˜ s t ) 7: ˜ s t +1 =
GetNextState( ˜ s t , a t ) 8: if q t +1 =
=
q f then 9: ˜ r t =(cid:46) terminal reward for satisfying φ 10: break (cid:46) φ is satisﬁed, restart episode 11: else if q t +1 =
= trap then 12: ˜ r t = −(cid:46) terminal reward for violating φ 13: break (cid:46) φ is violated, restart episode 14: else 15: ˜ r t =
GetReward( ˜ s t , ˜ s t +1 )
(cid:46) using Equation16: 17: end if 18:
B ← (˜ s t , a t , ˜ s t +1 , ˜ r t ) (cid:46) store experience in replay pool 19: if size ( B ) > N then 20: pop( B [0] ) 21: end if 22: θ ← UpdatePolicy( B ) (cid:46) this can be any RL update rule and doesn’t necessarily have to occur at this location 23: end for 24: end for
E S EMANTICS FOR SC TLTL
Following the syntax for scTLTL provided in Section 2.
[Equation]
here we deﬁne the semantics for the language.
We denote s t ∈ S to be the state at time t
[Equation]
and s t : t + k to be a sequence of states (state trajectory) from time t to t + k ,
i
[Equation]
e., s t : t + k = s t s t +1 ...
s t
+ k .
The Boolean semantics of scTLTL is deﬁned as:
[Equation]
Intuitively, s t : t + k | = ♦ φ if φ is satisﬁed for at least one sub-trajectory s t
[Equation]
: t + k , t (cid:48) ∈ [ t, t + k ) .
s t : t + k | = φ U ψ if φ is satisﬁed at every time step before ψ is satisﬁed, and ψ is satisﬁed at a time between t and t + k .
s t : t + k | = φ T ψ if φ is satisﬁed at least once before ψ is satisﬁed between t and t + k .
A trajectory s of duration k is said to satisfy formula φ if s 0: k | = φ .
We also deﬁne the quantitative semantics for scTLTL (robustness degree) , i.e., a real-valued function ρ ( s t : t + k , φ ) of state trajectory s t : t + k and a scTLTL speciﬁcation φ that indicates how far s t : t + k is from satisfying or violating the speciﬁcation φ .
The quantitative semantics of scTLTL is deﬁned as follows:
[Equation]
where ρ max represents the maximum robustness value.
Moreover, ρ ( s t : t + k , φ ) > 0 ⇒ s t : t + k | = φ and ρ ( s t : t + k , φ ) <
⇒ s t : t + k
(cid:54)| = φ , which implies that the robustness degree can substitute Boolean semantics in order to enforce the speciﬁcation φ .