Original: A UTOMATA G UIDED S KILL C OMPOSITION
Revised: A UTOMATA G UIDED S KILL C OMPOSITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Skills learned through(deep) reinforcement learning often generalizes poorly across tasks and re-training is necessary when presented with a new task.
Revised: Skills learned through reinforcement learning often generalize poorly across tasks and re-training is necessary when presented with a new task.

Original: We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for convenient speciﬁcation of complex temporal dependent tasks with logical expressions and construction of new skills from existing ones with no additional exploration.
Revised: We present a framework that combines techniques in formal methods with reinforcement learning (RL) that allows for the construction of new skills from existing ones with no additional exploration necessary. Our method also allows for convenient speciﬁcation of complex temporal dependent tasks using logical expressions.

Original: We provide theoretical results for our composition technique and evaluate on a simple grid world simulation as well as a robotic manipulation task.
Revised: We provide theoretical results for our composition technique and evaluate on a simple grid world simulation as well as a robotic manipulation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Policies learned using reinforcement learning aim to maximize the given reward function and is often difﬁcult to transfer to other problem domains.
Revised: Policies learned using reinforcement learning aim to maximize the given reward function and are often difﬁcult to transfer to other problem domains.

Original: Skill composition is the process of constructing new skills out of existing ones (policies) with little to no additional learning.
Revised: Skill composition is the process of constructing new skills out of existing ones (policies) with little to no additional learning.

Original: In stochastic optimal control, this idea has been adopted by authors of (Todorov, 2009) and (Da Silva et al., 2009) to construct provably optimal control laws based on linearly solvable Markov decision processes.
Revised: In stochastic optimal control, this idea has been adopted by Todorov (2009) and Da Silva et al. (2009) to construct provably optimal control laws based on linearly solvable Markov decision processes.

Original: Temporal logic (TL) is a formal language commonly used in software and digital circuit veriﬁcation (Baier & Katoen, 2008) as well as formal synthesis (Belta et al., 2017).
Revised: Temporal logic (TL) is a formal language commonly used in software and digital circuit veriﬁcation (Baier & Katoen, 2008) as well as formal synthesis (Belta et al., 2017).

Original: It allows for convenient expression of complex behaviors and causal relationships.
Revised: It allows for convenient expression of complex behaviors and causal relationships.

Original: TL has been used by (Tabuada & Pappas, 2004), (Fainekos et al., 2006), (Fainekos et al., 2005) to synthesize provably correct control policies.
Revised: TL has been used by (Tabuada & Pappas, 2004), (Fainekos et al., 2006), (Fainekos et al., 2005) to synthesize provably correct control policies.

Original: Authors of (Aksaray et al., 2016) have also combined TL with Q-learning to learn satisﬁable policies in discrete state and action spaces.
Revised: Aksaray et al. (2016) have also combined TL with Q-learning to learn satisﬁable policies in discrete state and action spaces.

Original: In this work, we focus on skill composition with policies learned using automata guided reinforcement learning (Li et al., 2018).
Revised: In this work, we focus on skill composition with policies learned using automata guided reinforcement learning (Li et al., 2018).

Original: We adopt the syntactically co-safe truncated linear temporal logic (scTLTL) as the task speciﬁcation language.
Revised: We adopt the syntactically co-safe truncated linear temporal logic (scTLTL) as the task speciﬁcation language.

Original: Compared to most heuristic reward structures used in the RL literature, formal speciﬁcation language has the advantage of semantic rigor and interpretability.
Revised: Compared to most heuristic reward structures used in the RL literature, formal speciﬁcation language has the advantage of semantic rigor and interpretability.

Revised: We make the distinction between skill composition and multi-task learning/meta-learning where the latter often requires a predeﬁned set of tasks/task distributions to learn and generalize from, whereas the focus of the former is to construct policies that achieve new tasks from a library of already learned policies with little to no additional constraints on task distribution at learning time.

Revised: In this work, we will mainly focus on skill composition.

Original: In our framework, skill composition is accomplished by taking the product of ﬁnite state automata (FSA).
Revised: In our framework, skill composition is accomplished by taking the product of ﬁnite state automata (FSA).

Original: Instead of interpolating/extrapolating among learned skills/latent features, our method is based on graph manipulation of the FSA.
Revised: Instead of interpolating/extrapolating among learned skills/latent features, our method is based on graph manipulation of the FSA.

Original: Therefore, thecompositional outcome is much more transparent.
Revised: Therefore, the outcome is much more transparent.

Original: At testing time, the behavior of the policy is strictly enforced by the FSA and therefore safety can be guaranteed if encoded in the speciﬁcation.

Original: Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class.
Revised: Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class.

Original: We validate our framework in simulation (discrete state and action spaces) and experimentally on a Baxter robot (continuous state and action spaces).
Revised: We validate our framework in simulation (discrete state and action spaces) and experimentally on a Baxter robot (continuous state and action spaces).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent efforts in skill composition have mainly adopted the approach of combining value functions learned using different rewards.
Revised: Recent efforts in skill composition have mainly adopted the approach of combining value functions learned using different rewards.

Original: Authors of (Peng et al., 2018) construct a composite policy by combining the value functions of individual policies using the Boltzmann distribution.
Revised: Peng et al. (2018) constructs a composite policy by combining the value functions of individual policies using the Boltzmann distribution.

Original: With a similar goal, authors of (Zhu et al., 2017) achieves task space transfer using deep successor representations (Kulkarni et al., 2016).
Revised: With a similar goal, Zhu et al. (2017) achieves task space transfer using deep successor representations (Kulkarni et al., 2016).

Original: However, it is required that the reward function be represented as a linear combination of state-action features.
Revised: However, it is required that the reward function be represented as a linear combination of state-action features.

Original: Authors of (Haarnoja et al., 2018) have showed that when using energy-based models (Haarnoja et al., 2017), an approximately optimal composite policy can result from taking the average of the Q-functions of existing policies.
Revised: Haarnoja et al. (2018) have showed that when using energy-based models (Haarnoja et al., 2017), an approximately optimal composite policy can result from taking the average of the Q-functions of existing policies.

Original: The resulting composite policy achieves the − AND − task composition i.e. the composite policy maximizes the average reward of individual tasks.
Revised: The resulting composite policy achieves the − AND − task composition i.e. the composite policy maximizes the average reward of individual tasks.

Original: Authors of (van Niekerk et al., 2018) have taken this idea a step further and showed that by combining individual Q-functions using the log-sum-exponential function, the − OR − task composition (the composite policy maximizes the (soft) maximum of the reward of constituent tasks) can be achieved optimally.
Revised: van Niekerk et al. (2018) have taken this idea a step further and showed that by combining individual Q-functions using the log-sum-exponential function, the − OR − task composition (the composite policy maximizes the (soft) maximum of the reward of constituent tasks) can be achieved optimally.

Original: We build on the results of (van Niekerk et al., 2018) and show that incorporating temporal logic allows us to compose tasks of greater logical complexity with higher interpretability.
Revised: We build on the results of (van Niekerk et al., 2018) and show that incorporating temporal logic allows us to compose tasks of greater logical complexity with higher interpretability.

Original: Our composite policy is optimal in both − AND − and − OR − task compositions.
Revised: Our composite policy is optimal in both − AND − and − OR − task compositions.

Original:  P RELIMINARIES 3. E NTROPY -R EGULARIZED R EINFORCEMENT L EARNING AND Q-C OMPOSITION
Revised:  P RELIMINARIES 3. E NTROPY -R EGULARIZED R EINFORCEMENT L EARNING AND Q-C OMPOSITION

Original: We start with the deﬁnition of a Markov Decision Process.
Revised: We start with the deﬁnition of a Markov Decision Process.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: An MDP is deﬁned as a tuple M = (cid:104) S, A, p ( ·|· , · ) , r ( · , · , · ) (cid:105) , where S ⊆
Revised: An MDP is deﬁned as a tuple M = (cid:104) S, A, p ( ·|· , · ) , r ( · , · , · ) (cid:105) , where S ⊆

Original: IR n is the state space ; A ⊆ IR m is the action space ( S and A can also be discrete sets); p : S ×
Revised: IR n is the state space ; A ⊆ IR m is the action space ( S and A can also be discrete sets); p : S ×

Original: A × S → [0 , 1] is the transition function with p ( s (cid:48) | s, a ) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s (cid:48) ∈ S ; r : S ×
Revised: A × S → [0 , 1] is the transition function with p ( s (cid:48) | s, a ) being the conditional probability density of taking action a ∈ A at state s ∈ S and ending up in state s (cid:48) ∈ S ; r : S ×

Original: A × S → IR is the reward function with r ( s, a, s (cid:48) ) being the reward obtained by executing action a at state s and transitioning to s (cid:48) .
Revised: A × S → IR is the reward function with r ( s, a, s (cid:48) ) being the reward obtained by executing action a at state s and transitioning to s (cid:48) .

Original: In entropy-regularized reinforcement learning (Schulman et al., 2017), the goal is to maximize the following objective
Revised: In entropy-regularized reinforcement learning (Schulman et al., 2017), the goal is to maximize the following objective

Original: [Equation]
Revised: [Equation]

Original: where π : S ×
Revised: where π : S ×

Original: A → [0 , 1] is a stochastic policy.
Revised: A → [0 , 1] is a stochastic policy.

Original: E π is the expectation following π . H ( π ( ·| s t )) is the entropy of π .
Revised: E π is the expectation following π . H ( π ( ·| s t )) is the entropy of π .

Original: α is the temperature parameter.
Revised: α is the temperature parameter.

Original: In the limit α → 0 , Equation (1) becomes the standard
Revised: In the limit α → 0 , Equation (1) becomes the standard

Original: RL objective.
Revised: RL objective.

Original: The soft Q-learning algorithm introduced by (Haarnoja et al., 2017) optimizes the above objective and ﬁnds a policy represented by an energy-based model
Revised: The soft Q-learning algorithm introduced by (Haarnoja et al., 2017) optimizes the above objective and ﬁnds a policy represented by an energy-based model

Original: [Equation]
Revised: [Equation]

Original: where E ( s t , a t ) is an energy function that can be represented by a function approximator.
Revised: where E ( s t , a t ) is an energy function that can be represented by a function approximator.

Original: Let r entt =
Revised: Let r entt =

Original: r t + α H
Revised: r t + α H

Original: ( π ( ·| s t )) , the state-action value function (Q-function) following π is deﬁned as Q π ( s, a ) =
Revised: ( π ( ·| s t )) , the state-action value function (Q-function) following π is deﬁned as Q π ( s, a ) =

Original: E π [ (cid:80) T − 1 t =

Revised: E π [ (cid:80) T − 1 t =

Original: r entt | s 0 =
Revised: r entt | s 0 =

Original: s, a 0 = a ] .
Revised: s, a 0 = a ] .

Original: Suppose we have a set of n tasks indexed by i, i ∈ { 0 , ..., n } , each task is deﬁned by an MDP M i that differs only in their reward function r i .
Revised: Suppose we have a set of n tasks indexed by i, i ∈ { 0 , ..., n } , each task is deﬁned by an MDP M i that differs only in their reward function r i .

Original: Let Q π (cid:63)i α be the optimal entropy-regularized Q-function.
Revised: Let Q π (cid:63)i α be the optimal entropy-regularized Q-function.

Original: Authors of (van Niekerk et al., 2018) provide the following results
Revised: Authors of (van Niekerk et al., 2018) provide the following results

Original: Theorem 1.
Revised: Theorem 1.

Original: Deﬁne vectors r =
Revised: Deﬁne vectors r =

Original: [ r 1 , ..., r n ] ,Q π (cid:63) α
Revised: [ r 1 , ..., r n ] ,

Original: [Equation]
Revised: [Equation]

Original: Given a set of nonnegative weights w with || w || = 1 , the optimal Q-function for a new task deﬁned by r = α log( || exp( r /α )
Revised: Given a set of nonnegative weights w with || w || = 1 , the optimal Q-function for a new task deﬁned by r = α log( || exp( r /α )

Original: || w ) is given by
Revised: || w ) is given by

Original: [Equation]
Revised: [Equation]

Original: 3) where || · || w is the weighted 1-norm.
Revised: where || · || w is the weighted 1-norm.

Original: The authors proceed to provide the following corollary
Revised: The authors proceed to provide the following corollary

Original: Corollary 1.
Revised: Corollary 1.

Revised: max Q

Original: max Q π (cid:63) α ↑ Q π (cid:63) 0 as α → 0 , where Q π (cid:63) 0 is the optimal Q-function for the objective
Revised: [Equation] α [Equation] as α → 0 , where Q π (cid:63) 0 is the optimal Q-function for the objective

Original: J ( π ) =
Revised: J ( π [Equation]

Original: [-(cid:80) T − 1 t =
-]
Revised: [-(cid:80) T − 1 t =
-]=

Original: E π [ r t ] .
Revised: [Equation] [Equation]

Original: Corollary 1 states that in the low temperature limit, the maximum of the optimal entropy-regularized
Revised: Corollary 1 states that in the low temperature limit, the maximum of the optimal entropy-regularized

Original: Q-functions approaches the standard optimal Q-function
Revised: Q-functions approaches the standard optimal Q-function

Original: 3. SC TLTL AND F INITE S TATE A UTOMATA
Revised: 3. SC TLTL AND F INITE S TATE A UTOMATA

Original: We consider tasks speciﬁed with syntactically co-safe Truncated Linear Temporal Logic (scTLTL) which is derived from truncated linear temporal logic(TLTL)
Revised: We consider tasks speciﬁed with syntactically co-safe Truncated Linear Temporal Logic (scTLTL) which is derived from truncated linear temporal logic(TLTL)

Original: (Li et al., 2018).
Revised: (Li et al., 2018).

Original: The syntax of scTLTL is deﬁned as
Revised: The syntax of scTLTL is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where (cid:62) is the True Boolean constant.
Revised: where (cid:62) is the True Boolean constant.

Original: s ∈ S is a MDP state in Deﬁnition 1.
Revised: s ∈ S is a MDP state in Deﬁnition 1.

Original: f ( s ) <
Revised: f ( s ) <

Original: c is a predicate over the MDP states where c ∈ IR .
Revised: c is a predicate over the MDP states where c ∈ IR .

Original: ¬ (negation/not), ∧ (conjunction/and) are Boolean connectives.
Revised: ¬ (negation/not), ∧ (conjunction/and) are Boolean connectives.

Original: ♦ (eventually), U (until), T (then), (cid:13) (next), are temporal operators.
Revised: ♦ (eventually), U (until), T (then), (cid:13) (next), are temporal operators.

Original: ⇒ (implication) and and ∨ (disjunction/or) can be derived from the above operators.
Revised: ⇒ (implication) and and ∨ (disjunction/or) can be derived from the above operators.

Original: We denote s t ∈ S to be the MDP state at time t , and s t : t + k to be a sequence of states (state trajectory) from time t to t + k , i.e., s t : t + k = s t s t
Revised: We denote s t ∈ S to be the MDP state at time t , and s t : t + k to be a sequence of states (state trajectory) from time t to t + k , i.e., s t : t + k = s t s t

Original: +1 ...
Revised: +1 ...

Original: s t
Revised: s t

Original: + k .
Revised: + k .

Original: The Boolean semantics of scTLTL is deﬁned as:
Revised: The Boolean semantics of scTLTL is deﬁned as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original:  The quantitative semantics (also referred to as robustness) is deﬁned recursively asρ [Equation] t : t +
Revised: A trajectory s 0: T is said to satisfy formula φ if s 0: T | = φ . The quantitative semantics (also referred to as robustness) is deﬁned recursively as

Original: [Equation]
Revised: [Equation]

Original: where ρ max represents the maximum robustness value.
Revised: where ρ max represents the maximum robustness value.

Original: A robustness of greater than zero implies that s t : t + k satisﬁes φ and vice versa ( ρ ( s t : t + k , φ ) > 

Revised: A robustness of greater than zero implies that s t : t + k satisﬁes φ and vice versa ( ρ ( s t : t + k , φ ) >

Original: ⇒ s t : t + k | = φ and ρ ( s t : t + k , φ ) < 

Revised: ⇒ s t : t + k | = φ and ρ ( s t : t + k , φ ) <

Original: ⇒ s t : t + k
Revised: ⇒ s t : t + k

Original: (cid:54)| = φ ).
Revised: (cid:54)| = φ ).

Original: The robustness is used as a measure of the level of satisfaction of a trajectory s 0: T with respect to a scTLTL formula φ .
Revised: The robustness is used as a measure of the level of satisfaction of a trajectory s 0: T with respect to a scTLTL formula φ .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: An FSA corresponding to a scTLTL formula φ . is deﬁned as a tuple A φ = (cid:104) Q φ , Ψ φ , q φ, 0 , p φ ( ·|· ) , F φ (cid:105) , where Q φ is a set of automaton states; Ψ φ is the input alphabet (a set of ﬁrst order logic formula); q φ, 0 ∈ Q φ is the initial state; p φ : Q φ × Q φ → [0 , 1] is a conditional probability deﬁned as[Equation] true
Revised: An FSA corresponding to a scTLTL formula φ . is deﬁned as a tuple A φ = (cid:104) Q φ , Ψ φ , q φ, 0 , p φ ( ·|· ) , F φ (cid:105) , where Q φ is a set of automaton states; Ψ φ is the input alphabet (a set of ﬁrst order logic formula); q φ, 0 ∈ Q φ is the initial state; p φ : Q φ × Q φ → [0 , 1] is a conditional probability deﬁned as

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: F φ is a set of ﬁnal automaton states.

Original: Here q φ,i is the i th automaton state of A φ .
Revised: Here q φ,i is the i th automaton state of A φ .

Original: ψ q φ,i ,q φ,j ∈ Ψ φ is the predicate guarding the transi- tion from q φ,i to q φ,j .
Revised: ψ q φ,i ,q φ,j ∈ Ψ φ is the predicate guarding the transi- tion from q φ,i to q φ,j .

Original: Because ψ q φ,i ,q φ,j is a predicate without temporal operators, the robustness ρ ( s t : t + k , ψ q φ,i ,q φ,j ) is only evaluated at s t .
Revised: Because ψ q φ,i ,q φ,j is a predicate without temporal operators, the robustness ρ ( s t : t + k , ψ q φ,i ,q φ,j ) is only evaluated at s t .

Original: Therefore, we use the shorthand ρ ( s t , ψ q φ,i ,q φ,j ) =
Revised: Therefore, we use the shorthand ρ ( s t , ψ q φ,i ,q φ,j ) =

Original: ρ ( s t : t + k , ψ q φ,i ,q φ,j ) .
Revised: ρ ( s t : t + k , ψ q φ,i ,q φ,j ) .

Original: The translation from a TLTL formula to a FSA can be done automatically with available packages like Lomap (Vasile, 2017).
Revised: The translation from a TLTL formula to a FSA can be done automatically with available packages like Lomap (Vasile, 2017).

Original: 3. FSA A UGMENTED MDP
Revised: 3. FSA A UGMENTED MDP

Original: The FSA Augmented MDP is deﬁned as follows
Revised: The FSA Augmented MDP is deﬁned as follows

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: (Li et al., 2018)
Revised: (Li et al., 2018)

Original: An FSA augmented MDP corresponding to scTLTL formula φ (constructed from FSA (cid:104) Q φ , Ψ φ , q φ, 0 , p φ ( ·|· ) , F φ (cid:105) and MDP (cid:104) S, A, p ( ·|· , · ) , r ( · , · , · ) (cid:105) ) is deﬁned as
Revised: An FSA augmented MDP corresponding to scTLTL formula φ (constructed from FSA (cid:104) Q φ , Ψ φ , q φ, 0 , p φ ( ·|· ) , F φ (cid:105) and MDP (cid:104) S, A, p ( ·|· , · ) , r ( · , · , · ) (cid:105) ) is deﬁned as

Original: M φ = (cid:104) ˜ S, A, ˜ p ( ·|· , · ) , ˜ r ( · , · ) , F φ (cid:105) where ˜ S ⊆
Revised: M φ = (cid:104) ˜ S, A, ˜ p ( ·|· , · ) , ˜ r ( · , · ) , F φ (cid:105) where ˜ S ⊆

Original: S × Q φ , ˜ p (˜ s (cid:48) | ˜ s, a ) is the probability of transitioning to ˜ s (cid:48) given ˜ s and a ,
Revised: S × Q φ , ˜ p (˜ s (cid:48) | ˜ s, a ) is the probability of transitioning to ˜ s (cid:48) given ˜ s and a ,

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: (6) p φ is deﬁned in Equation equation 5.
Revised: p φ is deﬁned in Equation equation 5.

Original: ˜ r : ˜ S × ˜ S → IR is the FSA augmented reward function, deﬁned by
Revised: ˜ r : ˜ S × ˜ S → IR is the FSA augmented reward function, deﬁned by

Original: [Equation]
Revised: [Equation]

Original: (7) where D q

Original: [Equation]

Original: = (cid:87)

Original: [Equation] qφ ψ [Equation] φ represents the disjunction of all predicates guarding the transitions that originate from q φ ( Ω q φ is the set of automata states that are connected with q through outgoing edges).
Revised: where D q φ φ = (cid:87) q (cid:48) φ ∈ Ω qφ ψ q φ ,q (cid:48) φ represents the disjunction of all predicates guarding the transitions that originate from q φ ( Ω q φ is the set of automata states that are connected with q through outgoing edges).

Original: A policy π (cid:63)φ is said to satisfy 
Revised: A policy π (cid:63)φ is said to satisfy φ if

Original: [Equation]
Revised: [Equation]

Original: where 1 ( ρ ( s 0: T , φ ) > 0) is an indicator function with value 1 if ρ ( s 0: T , φ ) > 0 and 0 otherwise.
Revised: where 1 ( ρ ( s 0: T , φ ) > 0) is an indicator function with value 1 if ρ ( s 0: T , φ ) > 0 and 0 otherwise.

Original: As is mentioned in the original paper, there can be multiple policies that meet the requirement of
Revised: As is mentioned in the original paper, there can be multiple policies that meet the requirement of

Original: Equation (8), therefore, a discount factor is used to ﬁnd a maximally satisfying policy - one that leads to satisfaction in the least amount of time.
Revised: Equation (8), therefore, a discount factor is used to ﬁnd a maximally satisfying policy - one that leads to satisfaction in the least amount of time.

Original: The FSA augmented MDP M φ establishes a connection between the TL speciﬁcation and the standard reinforcement learning problem.
Revised: The FSA augmented MDP M φ establishes a connection between the TL speciﬁcation and the standard reinforcement learning problem.

Original: A policy learned using M φ has implicit knowledge of the
Revised: A policy learned using M φ has implicit knowledge of the

Original: FSA through the automaton state q φ ∈ Q φ .
Revised: FSA through the automaton state q φ ∈ Q φ .

Original: We will take advantage of this characteristic in our skill composition framework.
Revised: We will take advantage of this characteristic in our skill composition framework.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: Problem 1.
Revised: Problem 1.

Original: Given two scTLTL formula φ 1 and φ 2 and their optimal Q-functions Q (cid:63)φ 1 and Q (cid:63)φ 2 , obtain the optimal policy π (cid:63)φ ∧ that satisﬁes φ ∧ = φ 1 ∧ φ 2 and π (cid:63)φ ∨ that satisﬁes φ ∨ = φ 1 ∨ φ 2 .
Revised: Given two scTLTL formula φ 1 and φ 2 and their optimal Q-functions Q (cid:63)φ 1 and Q (cid:63)φ 2 , obtain the optimal policy π (cid:63)φ ∧ that satisﬁes φ ∧ = φ 1 ∧ φ 2 and π (cid:63)φ ∨ that satisﬁes φ ∨ = φ 1 ∨ φ 2 .

Original: Here Q (cid:63)φ 1 and Q (cid:63)φ 2 can be the optimal Q-functions for the entropy-regularized MDP or the standard MDP.
Revised: Here Q (cid:63)φ 1 and Q (cid:63)φ 2 can be the optimal Q-functions for the entropy-regularized MDP or the standard MDP.

Original: Problem 1 deﬁnes the problem of skill composition: given two policies each satisfying a scTLTL speciﬁcation, construct the policy that satisﬁes the conjunction ( − AND − )/disjunction
Revised: Problem 1 deﬁnes the problem of skill composition: given two policies each satisfying a scTLTL speciﬁcation, construct the policy that satisﬁes the conjunction ( − AND − )/disjunction

Original: ( − OR − ) of the given speciﬁcations.
Revised: ( − OR − ) of the given speciﬁcations.

Original: Solving this problem is useful when we want to break a complex task into simple and manageable components, learn a policy that satisﬁes each component and ”stitch” all the components together so that the original task is satisﬁed.
Revised: Solving this problem is useful when we want to break a complex task into simple and manageable components, learn a policy that satisﬁes each component and ”stitch” all the components together so that the original task is satisﬁed.

Original: It can also be the case that as the scope of the task grows with time, the original task speciﬁcation is amended with new items.
Revised: It can also be the case that as the scope of the task grows with time, the original task speciﬁcation is amended with new items.

Original: Instead of having to re-learn the task from scratch, we can learn only policies that satisﬁes the new items and combine them with the old policy.
Revised: Instead of having to re-learn the task from scratch, we can learn only policies that satisﬁes the new items and combine them with the old policy.

Original:  A UTOMATA G UIDED S KILL C OMPOSITION
Revised:  A UTOMATA G UIDED S KILL C OMPOSITION

Original: In section, we provide a solution for Problem 1 by constructing the FSA of φ ∧ from that of φ 1 and φand using φ ∧ to synthesize the policy for the combined skill.
Revised: In section, we provide a solution for Problem 1 by constructing the FSA of φ ∧ from that of φ 1 and φand using φ ∧ to synthesize the policy for the combined skill.

Original: We start with the following deﬁnition.
Revised: We start with the following deﬁnition.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: Given A φ 1 = (cid:104) Q φ 1 , Ψ φ 1 , q φ 1 , 0 , p φ 1 , F φ 1 (cid:105) and A φ 2 = (cid:104) Q φ 2 , Ψ φ 2 , q φ 2 , 0 , p φ 2 , F φ 2 (cid:105)
Revised: Given A φ 1 = (cid:104) Q φ 1 , Ψ φ 1 , q φ 1 , 0 , p φ 1 , F φ 1 (cid:105) and A φ 2 = (cid:104) Q φ 2 , Ψ φ 2 , q φ 2 , 0 , p φ 2 , F φ 2 (cid:105)

Original: corresponding to formulas φ 1 and φ 2 , the FSA of φ ∧ = φ 1 ∧ φ 2 is the product automaton of A φ 1 and
Revised: corresponding to formulas φ 1 and φ 2 , the FSA of φ ∧ = φ 1 ∧ φ 2 is the product automaton of A φ 1 and

Original: A φ 1 , i.e. A φ ∧ = φ 1 ∧ φ 2 =
Revised: A φ 1 , i.e. A φ ∧ = φ 1 ∧ φ 2 =

Original: A φ 1 × A φ 2 = (cid:104) Q φ ∧ , Ψ φ ∧ , q φ ∧ , 0 , p φ ∧ , F φ ∧ (cid:105) where Q φ ∧ ⊆ Q φ 1 × Q φ 2 is the set of product automaton states, q φ ∧ , 0 =
Revised: A φ 1 × A φ 2 = (cid:104) Q φ ∧ , Ψ φ ∧ , q φ ∧ , 0 , p φ ∧ , F φ ∧ (cid:105) where Q φ ∧ ⊆ Q φ 1 × Q φ 2 is the set of product automaton states, q φ ∧ , 0 =

Original: ( q φ 1 , 0 , q φ 2 , 0 ) is the product initial state,
Revised: ( q φ 1 , 0 , q φ 2 , 0 ) is the product initial state,

Original: F φ ∧ ⊆
Revised: F φ ∧ ⊆

Original: F φ 1 ∩ F φ 2 are the ﬁnal accepting states.
Revised: F φ 1 ∩ F φ 2 are the ﬁnal accepting states.

Original: Following Deﬁnition 2, for states q φ ∧ = ( q φ 1 , q φ 2 ) ∈ Q φ ∧ and q (cid:48) [Equation] , q [Equation]
Revised: Following Deﬁnition 2, for states q φ ∧ = ( q φ 1 , q φ 2 ) ∈ Q φ ∧ and q (cid:48) φ ∧ = ( q (cid:48) φ 1 , q (cid:48)

Original: [Equation]
Revised: [Equation]

Original:  ∈ Q φ ∧ , the transition probability p φ ∧ is deﬁned as
Revised: 2 ) ∈ Q φ ∧ , the transition probability p φ ∧ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Example 1.
Revised: Example 1.

Original: Figure 1 illustrates the FSA of A φ 1 and A φ 2 and their product automaton
Revised: Figure 2 illustrates the FSA of A φ 1 and A φ 2 and their product automaton

Original: A φ ∧ .
Revised: A φ ∧ .

Original: Here φ 1 = ♦
Revised: Here φ 1 = ♦

Original: r ∧♦ g which entails that both r and g needs to be true at least once (order does not matter), and φ 2 = ♦ b .
Revised: r ∧♦ g which entails that both r and g needs to be true at least once (order does not matter), and φ 2 = ♦ b .

Original: The resultant product corresponds to the formula φ = ♦ r ∧ ♦ g ∧ ♦ b .
Revised: The resultant product corresponds to the formula φ = ♦ r ∧ ♦ g ∧ ♦ b .

Original: We provide the following theorem on automata guided skill composition
Revised: We provide the following theorem on automata guided skill composition

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Q π (cid:63) φ =
Revised: Let Q π (cid:63) φ =

Original: [ Q π (cid:63) 1 φ 1 , ..., Q π (cid:63)n φ n ] be a vector with entries Q π (cid:63)i φ i being the optimal Q-function for the FSA augmented MDP M φ i .
Revised: [ Q π (cid:63) 1 φ 1 , ..., Q π (cid:63)n φ n ] be a vector with entries Q π (cid:63)i φ i being the optimal Q-function for the FSA augmented MDP M φ i .

Original: The optimal Q-function for M φ ∧ where φ ∧ = (cid:86) i φ i is Q π (cid:63) ∧ φ ∧ = max( Q π (cid:63) φ ) .
Revised: The optimal Q-function for M φ ∧ where φ ∧ = (cid:86) i φ i is Q π (cid:63) ∧ φ ∧ = max( Q π (cid:63) φ ) .

Original: Proof.
Revised: Proof.

Original: For q φ ∧ = ( q φ 1 , q φ 2 ) ∈ Q φ ∧ , let Ψ q φ ∧ , Ψ
Revised: For q φ ∧ = ( q φ 1 , q φ 2 ) ∈ Q φ ∧ , let Ψ q φ ∧ , Ψ

Original: q φ 1 and Ψ q φ 2 denote the set of predicates guard-
Revised: q φ 1 and Ψ q φ 2 denote the set of predicates guard-

Original: ing the edges originating from q φ ∧ , q φ 1 and q φ 2 respectively.
Revised: ing the edges originating from q φ ∧ , q φ 1 and q φ 2 respectively.

Original: Equation equation 9 entails that a transition at q φ ∧ in the product automaton A φ ∧ exists only if corresponding transitions at q φ 1 , q φ 2 exist in A φ 1 and A φ 2 respectively.
Revised: Equation equation 9 entails that a transition at q φ ∧ in the product automaton A φ ∧ exists only if corresponding transitions at q φ 1 , q φ 2 exist in A φ 1 and A φ 2 respectively.

Original: Therefore, ψ q φ ∧ ,q (cid:48)
Revised: Therefore, ψ q φ ∧ ,q (cid:48)

Original: φ ∧ = ψ q φ 1 ,q (cid:48)
Revised: φ ∧ = ψ q φ 1 ,q (cid:48)

Original: φ 1 ∧ ψ q φ 2 ,q (cid:48) φ 2 , for ψ q φ ∧ ,q (cid:48) φ ∧ ∈ Ψ q φ ∧ , ψ q φ 1 ,q (cid:48)
Revised: φ 1 ∧ ψ q φ 2 ,q (cid:48) φ 2 , for ψ q φ ∧ ,q (cid:48) φ ∧ ∈ Ψ [Equation]

Original: φ 1 ∈ Ψ
Revised: [Equation] , ψ

Original: q φ 1 , ψ q φ 2 ,q (cid:48)
Revised: [Equation]

Original: φ 2 ∈ Ψ
Revised: ∈ Ψ

Original: q φ 2 (here q (cid:48) φ i is a state such that p φ i ( q (cid:48)
Revised: [Equation] 2 (here q (cid:48) φ i is a state such that p φ i ( q (cid:48)

Original: φ i | q φ i ) = 1 ).
Revised: φ i | q φ i ) = 1 ).

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original:  where q (cid:48)
Revised: (10) where q (cid:48)

Original: φ 1 , q (cid:48) φ 2 don’t equal to q φ 1 , q φ 2 at the same time (to avoid self looping edges).
Revised: φ 1 , q (cid:48) φ 2 don’t equal to q φ 1 , q φ 2 at the same time (to avoid self looping edges).

Original: Using the fact that ψ q φ i ,q φ i = ¬(cid:87) q
Revised: Using the fact that ψ q φi ,q φi = ¬

Original: [Equation]
Revised: [Equation]

Original: φi
Revised: ψ q

Original: [Equation]
Revised: [Equation]

Original: = q φi ψ q φ

Original: i ,q
Revised: ,q

Original: [Equation]
Revised: [Equation]

Original: φ i and repeatedly applying the distributive laws (∆ ∧ Ω 1 ) ∨ (∆ ∧ Ω 2 ) =
Revised: φi and repeatedly applying the distributive laws (∆ ∧ Ω 1 ) ∨ (∆ ∧ Ω 2 ) =

Original: ∆ ∧ (Ω 1 ∨ Ω 2 ) and (∆ ∨ Ω 1 ) ∧ (∆ ∨ Ω 2 ) =
Revised: ∆ ∧ (Ω 1 ∨ Ω 2 ) and (∆ ∨ Ω 1 ) ∧ (∆ ∨ Ω 2 ) =

Original: ∆ ∨ (Ω 1 ∧ Ω 2 ) to D q φ ∧ φ ∧ , we arrive at
Revised: ∆ ∨ (Ω 1 ∧ Ω 2 ) to D q φ ∧ φ ∧ , we arrive at

Original: [Equation]
Revised: [Equation]

Original:  Let ˜ r φ ∧ , ˜ r φ 1 , ˜ r φ 2 and ˜ s φ ∧ , ˜ s φ 1 , ˜ s φ 2 be the reward functions and states for FSA augmented MDP M φ ∧ , M φ 1 , M φ 2 respectively.
Revised: [Equation] Let ˜ r φ ∧ , ˜ r φ 1 , ˜ r φ 2 and ˜ s φ ∧ , ˜ s φ 1 , ˜ s φ 2 be the reward functions and states for FSA augmented MDP M φ ∧ , M φ 1 , M φ 2 respectively.

Original: s φ ∧ , s φ 1 , s φ 2 are the states for the corresponding MDPs.
Revised: s φ ∧ , s φ 1 , s φ 2 are the states for the corresponding MDPs.

Original: Plugging Equation (11) into Equation (7) and using the robustness deﬁnition for disjunction results in
Revised: Plugging Equation (11) into Equation (7) and using the robustness deﬁnition for disjunction results in

Original: [Equation]
Revised: [Equation]

Original: Looking at Theorem 1, the log-sum-exp of the composite reward r = α log( || exp( r /α )
Revised: Looking at Theorem 1, the log-sum-exp of the composite reward r = α log( || exp( r /α )

Original: || w ) is in fact an approximation of the maximum function.
Revised: || w ) is in fact an approximation of the maximum function.

Original: In the low temperature limit we have r → max( r ) as α → 0 .
Revised: In the low temperature limit we have r → max( r ) as α → 0 .

Original: Applying Corollary 1 results in Theorem 2.
Revised: Applying Corollary 1 results in Theorem 2.

Original: Having obtained the optimal Q-function, a policy can be constructed by taking the greedy step with respective to the Q-function in the discrete action case.
Revised: Having obtained the optimal Q-function, a policy can be constructed by taking the greedy step with respective to the Q-function in the discrete action case.

Original: For the case of continuous action space where the policy is represented by a function approximator, the policy update procedure in actorcritic methods can be used to extract a policy from the Q-function.
Revised: For the case of continuous action space where the policy is represented by a function approximator, the policy update procedure in actorcritic methods can be used to extract a policy from the Q-function.

Original: In our framework, − AND − and − OR − task compositions follow the same procedure (Theorem 2).
Revised: In our framework, − AND − and − OR − task compositions follow the same procedure (Theorem 2).

Original: The only difference is the termination condition.
Revised: The only difference is the termination condition.

Original: For − AND − task, the ﬁnal state F φ ∧ = (cid:84) F φ i in Deﬁnition 4 needs to be reached (i.e. all the constituent FSAs are required to reach terminal state, as in state q φ ∧ ,f in Figure 1 ).
Revised: For − AND − task, the ﬁnal state F φ ∧ = (cid:84) F φ i in Deﬁnition 4 needs to be reached (i.e. all the constituent FSAs are required to reach terminal state, as in state q φ ∧ ,f in Figure 2 ).

Original: Whereas for the − OR − task, only F φ ∨ = (cid:83) F φ i needs to be reached (one of states q φ ∧ , 2 , q φ ∧ , 4 , q φ ∧ , 5 , q φ ∧ , 6 , q φ ∧ ,f in Figure 1 ). 
Revised: Whereas for the − OR − task, only F φ ∨ = (cid:83) F φ i needs to be reached (one of states q φ ∧ , 2 , q φ ∧ , 4 , q φ ∧ , 5 , q φ ∧ , 6 , q φ ∧ ,f in Figure 2 ). A summary of the composition procedure is provided in Algorithm 1. [Table] [Table]

Original: C ASE S TUDIES
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: In Algorithm 1, steps 3 and 4 seeks to obtain the optimal policy and Q-function using any off-policy actor critic RL algorithm.

Revised: B φ 1 , 2 are the replay buffers collected while training for each skill.

Revised: Step 6 extracts the optimal composed policy from the optimal composed Q-function (this corresponds to running only the policy update step in the actor critic algorithm).

Revised:  C ASE S TUDIES

Original: We evaluate the our composition method in two environments.
Revised: We evaluate the our composition method in two environments.

Original: The ﬁrst is a simple 2D grid world environment that is used as for proof of concept and policy visualization.
Revised: The ﬁrst is a simple 2D grid world environment that is used as for proof of concept and policy visualization.

Original: The second is a robot manipulation environment.
Revised: The second is a robot manipulation environment

Original:  6. G RID W ORLD
Revised: . 6. G RID W ORLD

Original: Consider an agent that navigates in a 8 × 10 grid world.
Revised: Consider an agent that navigates in a 8 × 10 grid world.

Original: Its MDP state space is S :
Revised: Its MDP state space is S :

Original: X × Y where x, y ∈ X, Y are its integer coordinates on the grid.
Revised: X × Y where x, y ∈ X, Y are its integer coordinates on the grid.

Original: The action space is A : [ up , down , left , right , stay ].
Revised: The action space is A : [ up , down , left , right , stay ].

Original: The transition is such that for each action command, the agent follows that command with probability 0.8 or chooses a random action with probability 0.2.
Revised: The transition is such that for each action command, the agent follows that command with probability 0.8 or chooses a random action with probability 0.2.

Original: We train the agent on two tasks, φ 1 = ♦
Revised: We train the agent on two tasks, φ 1 = ♦

Original: r ∧ ♦ g and φ 2 = ♦ b (same as in Example 1).
Revised: r ∧ ♦ g and φ 2 = ♦ b (same as in Example 1).

Original: The regions are deﬁned by the predicates r = (1 < x < 3) ∧ (1 < y < 3) and g = (4 < x < 6) ∧ (4 < y < 6) .
Revised: The regions are deﬁned by the predicates r = (1 < x < 3) ∧ (1 < y < 3) and g = (4 < x < 6) ∧ (4 < y < 6) .

Original: Because the coordinates are integers, a and b deﬁne a point goal rather than regions.
Revised: Because the coordinates are integers, a and b deﬁne a point goal rather than regions.

Original: φ 2 expresses a similar task for b =
Revised: φ 2 expresses a similar task for b =

Original: (1 < x < 3) ∧ (6 < y < 8) .
Revised: (1 < x < 3) ∧ (6 < y < 8) .

Original: Figure 1 shows the FSA for each task.
Revised: Figure 2 shows the FSA for each task.

Original: We apply standard tabular Q-learning Watkins (1989) on the FSA augmented MDP of this environment.
Revised: We apply standard tabular Q-learning Watkins (1989) on the FSA augmented MDP of this environment.

Original: For all experiments, we use a discount factor of 0.95, learning rate of 0.1, episode horizon of 200 steps, a random exploration policy and a total number of 2000 update steps which is enough to reach convergence (learning curve is not presented here as it is not the focus of this paper).
Revised: For all experiments, we use a discount factor of 0.95, learning rate of 0.1, episode horizon of 200 steps, a random exploration policy and a total number of 2000 update steps which is enough to reach convergence (learning curve is not presented here as it is not the focus of this paper).

Original: Figure(a) and (b) show the learned optimal policies extracted by π (cid:63)φ i ( x, y, q φ i ) =
Revised: Figure ?? (a) and (b) show the learned optimal policies extracted by π (cid:63)φ i ( x, y, q φ i ) =

Original: arg max a
Revised: arg max a

Original: Q (cid:63)φ i ( x, y, q φ i , a ) .
Revised: Q (cid:63)φ i ( x, y, q φ i , a ) .

Original: We plot π (cid:63)φ i ( x, y, q φ i ) for each q φ i and observe that each represents a sub-policy whose goal is given by Equation 7.
Revised: We plot π (cid:63)φ i ( x, y, q φ i ) for each q φ i and observe that each represents a sub-policy whose goal is given by Equation 7.

Original: Figure 2 (c) shows the composed policy of φ ∧ = φ 1 ∧ φ 2 using Theorem 2.
Revised: Figure ?? (c) shows the composed policy of φ ∧ = φ 1 ∧ φ 2 using Theorem 2.

Original: It can be observed that the composed policy is able to act optimally in terms maximizing the expected sum of discounted rewards given by Equation (12).
Revised: It can be observed that the composed policy is able to act optimally in terms maximizing the expected sum of discounted rewards given by Equation (12).

Original: Following the composed policy and transitioning the FSA in Figure 1 (b) will in fact satisfy φ ∧ ( − AND − ).
Revised: Following the composed policy and transitioning the FSA in Figure 2 (b) will in fact satisfy φ ∧ ( − AND − ).

Original: As discussed in the previous section, if the − OR − task is desired, following the same composed policy and terminate at any of the states q φ ∧ , 2 , q φ ∧ , 4 , q φ ∧ , 5 , q φ ∧ , 6 , q φ ∧ ,f will satisfy φ ∨ = φ 1 ∨ φ 2 .
Revised: As discussed in the previous section, if the − OR − task is desired, following the same composed policy and terminate at any of the states q φ ∧ , 2 , q φ ∧ , 4 , q φ ∧ , 5 , q φ ∧ , 6 , q φ ∧ ,f will satisfy φ ∨ = φ 1 ∨ φ 2 .

Original: 6. R OBOTIC M ANIPULATION
Revised: 6. R OBOTIC M ANIPULATION

Original: In this sub-section, we test our method on a more complex manipulation task.
Revised: In this sub-section, we test our method on a more complex manipulation task.

Original: 6.2. E XPERIMENT S ETUP
Revised: 6.2. E XPERIMENT S ETUP

Original: Figure 4 (a) presents our experiment setup.
Revised: Figure 4 (a) presents our experiment setup.

Original: Our policy controls the 7 degree-of-freedom joint velocities of the right arm of a Baxter robot.
Revised: Our policy controls the 7 degree-of-freedom joint velocities of the right arm of a Baxter robot.

Original: In front of the robot are three circular regions (red, green, blue plates) are it has to learn to traverse in user speciﬁed ways.
Revised: In front of the robot are three circular regions (red, green, blue plates) are it has to learn to traverse in user speciﬁed ways.

Original: The positions of the plates are tracked by motion capture systems and thus fully observable.
Revised: The positions of the plates are tracked by motion capture systems and thus fully observable.

Original: In addition, we also track the position of one of the user’s hands (by wearing a glove with trackers attached).
Revised: In addition, we also track the position of one of the user’s hands (by wearing a glove with trackers attached).

Original: Our MDP state space isdimensional that includes 7 joint angles, xyz positions of the three regions (denoted by p r , p g , p b ), the user’s hand ( p h ) and the robot’s end-effector ( p ee ). 
Revised: Our MDP state space isdimensional that includes 7 joint angles, xyz positions of the three regions (denoted by p r , p g , p b ), the user’s hand ( p h ) and the robot’s end-effector ( p ee ). State and action spaces are continuous in this case.

Original: We deﬁne the following predicates
Revised: We deﬁne the following predicates

Original: ψ i = ||
Revised: ψ i = ||

Original: p i − p
Revised: p i − p

Original: ee
Revised: ee

Original: || < (cid:15), i ∈ { r, g, b, h } where (cid:15) is a threshold which we set to becentimeters.
Revised: || < (cid:15), i ∈ { r, g, b, h } where (cid:15) is a threshold which we set to becentimeters.

Original: ψ i constrains the relative distance between the robot’s end-effector and the selected object
Revised: ψ i constrains the relative distance between the robot’s end-effector and the selected object.

Original: ψ hand in sight = ( x min < p hx < x max ) ∧ ( y min <
Revised: ψ hand in sight = ( x min < p hx < x max ) ∧ ( y min <

Original: p hy < y max ) ∧ ( z min < p hz < z max ) .
Revised: p hy < y max ) ∧ ( z min < p hz < z max ) .

Original: This predicate evaluates to true if the user’s hand appears in the cubic region deﬁned by [ x min , x max , y min ,y max , z min , z max ] .
Revised: This predicate evaluates to true if the user’s hand appears in the cubic region deﬁned by [ x min , x max , y min ,

Original:  In this experiment, we take this region to becentimeters above the table (length and width the same as the table).
Revised: y max , z min z max ] . In this experiment, we take this region to becentimeters above the table (length and width the same as the table).

Original: We test our method on the following composition task
Revised: We test our method on the following composition task

Original: φ traverse = ♦ ( ψ r ∧ ♦ ( ψ g ∧ ♦ ψ b ))
Revised: φ traverse = ♦ ( ψ r ∧ ♦ ( ψ g ∧ ♦ ψ b ))

Original: Description: traverse the three regions in the order of red, green, blue. 2.
Revised: Description: traverse the three regions in the order of red, green, blue. 2.

Original: φ interrupt =
Revised: φ interrupt =

Original: ( ψ hand in sight ⇒ ♦ ψ h )
Revised: ( ψ hand in sight ⇒ ♦ ψ h )

Original: U ψ b Description: before reaching the blue region, if the user’s hand appears in sight, then eventually reach for the user’s hand and the blue region, otherwise just reach for the blue region.
Revised: U ψ b Description: before reaching the blue region, if the user’s hand appears in sight, then eventually reach for the user’s hand and the blue region, otherwise just reach for the blue region.

Original: φ ∧ = φ traverse ∧ φ interrupt Description: conjunction of the ﬁrst two tasks.
Revised: φ ∧ = φ traverse ∧ φ interrupt Description: conjunction of the ﬁrst two tasks.

Original: The FSAs for φ 1 and φ 2 are presented in Figure 3 .
Revised: The FSAs for φ 1 and φ 2 are presented in Figure 3 .

Original: The FSA for φ ∧ (14 nodes and 72 edges) is not presented here due to space constraints.
Revised: The FSA for φ ∧ (14 nodes and 72 edges) is not presented here due to space constraints.

Original: 6.2. I MPLEMENTATION D ETAILS
Revised: 6.2. I MPLEMENTATION D ETAILS

Original: Our policy and Q-function are represented by a neural network (3 layers each with 100 ReLU units).
Revised: Our policy and Q-function are represented by a neural network (3 layers each with 300, 200, ReLU units).

Original: For tasks φ traverse and φ interrupt , the input state space is 23 dimensional (22 continuous dimensional MDP state and 1 discrete dimension for the automaton state).
Revised: For tasks φ traverse and φ interrupt , the input state space is 23 dimensional (22 continuous dimensional MDP state and 1 discrete dimension for the automaton state).

Original: For task φ ∧ , the state space is 24 dimensional (2 discrete dimensions for automaton states of φ traverse and φ interrupt ).
Revised: For task φ ∧ , the state space is 24 dimensional (2 discrete dimensions for automaton states of φ traverse and φ interrupt ).

Original: We use soft Q-learning (SQL) (Haarnoja et al., 2017) to learn the optimal policies and Q-functions for φ traverse and φ interrupt and Theorem 2 to compose the Q-functions.
Revised: We use soft Q-learning (SQL) (Haarnoja et al., 2017) to learn the optimal policies and Q-functions for φ traverse and φ interrupt and Theorem 2 to compose the Q-functions.

Original: The policy for task φ ∧ is extracted from the composite Q-function by running the same algorithm on already collected experience without updating the Q-function (similar procedure as (Haarnoja et al., 2018)).
Revised: The policy for task φ ∧ is extracted from the composite Q-function by running the same algorithm on already collected experience without updating the Q-function (similar procedure as (Haarnoja et al., 2018)).

Original: All of our training is performed in simulation and the policy is able to transfer to the real robot without further ﬁne-tuning.

Original: After each episode, the joint angles,  position of the plates as well as  position of the hand (represented by the yellow sphere in Figure 4 (a)) are randomly reset (within certain boundaries) to ensure generalization across different task conﬁgurations.
Revised: After each episode, the joint angles, the FSA state, the position of the plates as well as the position of the hand (represented by the yellow sphere in Figure 4 (a)) are randomly reset (within certain boundaries) to ensure generalization across different task conﬁgurations.

Original: The robot is controlled at 20 Hz.
Revised: The robot is controlled at 20 Hz.

Original: Each episode is 100 time-steps (about 5 seconds).
Revised: Each episode is 100 time-steps (about 5 seconds).

Revised: The episode restarts if the ﬁnal automaton state q f is reached.

Revised: During training, we perform 100 policy and Q-function updates every 5 episodes of exploration.

Revised: All of our training is performed in simulation and for this set of tasks, the policy is able to transfer to the real robot without further ﬁne-tuning.

Original: 6.2. R ESULTS AND D ISCUSSION
Revised: 6.2. R ESULTS AND D ISCUSSION

Original: In Figurewe report theaverage return (sum of discounted rewards averaged over 5 episodes) as a function of  update steps for task φ ∧ .
Revised: In Figure 5 (left), we report the discounted return as a function of policy update steps for task φ ∧ .

Original: Since skill composition does not need any exploration, 5 evaluation episodes are collected after each update of the policy  to calculate the average return.
Revised: 5 evaluation episodes are collected after each set policy updates to calculate the performance statistics.

Original: As comparison, wealso learn the same taskfrom scratch using SQL.
Revised: As comparison, we learn the same task using SQL with FSA augmented MDP.

Original: We can see that our composition method takesmuch less update steps to reach a satisfying policy.
Revised: We can see that our composition method takes less update steps to reach a policy that achieves higher returns with lower variance than the policy obtained from learning.

Revised: Figure 5 (right) shows the episode length as a function of policy update (upper bound clipped at 100 steps).

Revised: As mentioned in the previous section, a shorter episode length indicates faster accomplishment of the task.

Revised: It can be observed that both the composition and learning method result in high variances likely due to the randomized task conﬁguration (some plate/joint/hand conﬁgurations make the task easier to accomplish than others).

Revised: However, the policy obtained from composition achieves a noticeable decrease in the average episode length.

Original: It is important to note that the wall time for learning a policyfrom scratch is signiﬁcantly longer than that from composition.
Revised: It is important to note that the wall time for learning a policy is signiﬁcantly longer than that from composition.

Original: For robotic tasks with relatively simple policy representations, learning speed is dominated by the time  to collect experience.
Revised: For robotic tasks with relatively simple policy representations (feed-forward neural networks), learning time is dominated by the time used to collect experiences and the average episode length (recall that we update the policy 100 times with each 5 episodes of exploration).

Original: Since skill composition usesthe already collected experience, obtaining a policy can be much faster.
Revised: Since skill composition uses already collected experience, obtaining a policy can be much faster.

Original: Table 1 shows the average time to convergence (over 5 random seeds) for each task.
Revised: Table 1 shows the mean training time and standard deviation (over 5 random seeds) for each task (tasks φ traverse , φ interrupt and φ ∧ ( learned ) are trained for 80K policy updates. φ ∧ ( composed ) is trained

Original: [Table]
Revised: [Table]

Revised: for 40K policy updates).

Revised: In general, training time is shorter for tasks with higher episodic success rate and shorter episode length.

Original: In Table 1 we also show theaverage task success rate evaluated on the  robot over 20  trials.
Revised: We also show the task success rate evaluated on the real robot over 20 evaluation trials.

Original: Task success is evaluated by calculating the robustness of the trajectories resulting from  each policy.
Revised: Task success is evaluated by calculating the robustness of the trajectories resulting from executing each policy.

Original: A robustness of greater than 0 evaluates to success and vice versa.
Revised: A robustness of greater than 0 evaluates to success and vice versa.

Original: π φ ∧  learned from scratch fails to complete the task even though a convergence is reached.
Revised: π φ ∧ ( learned ) fails to complete the task even though a convergence is reached during training.

Original: This is likely due to the large FSA  with complex per-step reward ( D qφ in Equation (7))that results from φ ∧ which makes learning difﬁcult.
Revised: This is likely due to the large FSA of φ ∧ with complex per-step reward ( D qφ in Equation (7)) which makes learning difﬁcult.

Original: Figure 4 (b) shows an evaluation run of the composed policy for task φ ∧ .
Revised: Figure 4 (b) shows an evaluation run of the composed policy for task φ ∧ .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We provide a technique that takes advantage of the product of ﬁnite state automata to perform deterministic skill composition.
Revised: We provide a technique that takes advantage of the product of ﬁnite state automata to perform deterministic skill composition.

Original: Our method is able to synthesize optimal composite policies for − AND − and − OR − tasks.
Revised: Our method is able to synthesize optimal composite policies for − AND − and − OR − tasks.

Original: We provide theoretical results on our method and show its effectiveness on a grid world simulation and a real world robotic task.
Revised: We provide theoretical results on our method and show its effectiveness on a grid world simulation and a real world robotic task.

Original: For future work, we will adapt our method to the more general case of task-space transfer - given a library of optimal policies (Qfunctions) that each satisﬁes its own speciﬁcation, construct a policy that satisﬁes a speciﬁcation that’s an arbitrary (temporal) logical combination of the constituent speciﬁcations.
Revised: For future work, we will adapt our method to the more general case of task-space transfer - given a library of optimal policies (Qfunctions) that each satisﬁes its own speciﬁcation, construct a policy that satisﬁes a speciﬁcation that’s an arbitrary (temporal) logical combination of the constituent speciﬁcations.