Robust Option Learning for Compositional Generalization
Abstract
Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks.
Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform each subtask.
In this paper, we focus on the problem of training subtask policies in a way that they can be used to perform any task; here, a task is given by a sequence of subtasks.
We aim to maximize the worst-case performance over all tasks as opposed to the average-case performance.
We formulate the problem as a two agent zero-sum game in which the adversary picks the sequence of subtasks.
We propose two RL algorithms to solve this game: one is an adaptation of existing multi-agent RL algorithms to our setting and the other is an asynchronous version which enables parallel training of subtask policies.
We evaluate our approach on two multi-task environments with continuous states and actions and demonstrate that our algorithms outperform state-of-the-art baselines.
Introduction
Reinforcement learning (RL) has proven to be a promising strategy for solving complex control tasks such as walking [13], autonomous driving [17], and dexterous manipulation [3].
However, a key challenge facing the deployment of reinforcement learning in real-world tasks is its high sample complexity—to solve any new task requires a training a new policy designed to solve that task.
One promising solution is compositional reinforcement learning , where individual options (or skills ) are first trained to solve simple tasks; then, these options can be composed together to solve more complex tasks [25, 24, 17].
For example, if a driving robot learns how to make left and right turns and to drive in a straight line, it can then drive along any route composed of these primitives.
A key challenge facing compositional reinforcement learning is the generalizability of the learned options.
In particular, options trained under one distribution of tasks may no longer work well if used in a new task, since the distribution of initial states from which the options are used may shift.
An alternate approach is to train the options separately to perform specific subtasks, but options trained this way might cause the system to reach states from which future subtasks are hard to perform.
One can overcome this issue by handcrafting rewards to encourage avoiding such states [17], in which case they generalize well, but this approach relies heavily on human time and expertise.
We propose a novel framework that addresses this challenges by formulating the option learning problem as an adversarial reinforcement learning problem.
At a high level, the adversary chooses the task that minimizes the reward achieved by composing the available options.
Thus, the goal is to learn a set of robust options that perform well across all potential tasks.
Then, we provide two algorithms for solving this problem.
The first adapts existing ideas for using reinforcement learning
[Figure]
to solve Markov games to our setting. Then, the second shows how to leverage the compositional structure of our problem to learn options in parallel at each step of a value iteration procedure; in some cases, by enabling such parallelism, we can reduce the computational cost of learning.
We validate our approach on two benchmarks: (i) a rooms environment where a point mass robot must navigate any given sequence of rooms, where the sequence is an arbitrary combination of straight, left, and right turns, and (ii) a simulated version of the F1/10th car, where a small racing car must navigate any racetrack composed of several different track segments.
In both, our empirical results demonstrate that robust options are critical for performing well on a wide variety of tasks.
In summary, our contributions are: (i) a game theoretic formulation of the compositional reinforcement learning problem, (ii) two algorithms for solving this problem, and (iii) an empirical evaluation demonstrating the effectiveness of our approach.
Motivating example.
Let us consider a small scale autonomous racing car shown in Figure 1 (a).
We would like to train a controller that can be used to navigate the car through all tracks constructed using five kinds of segments; the possible segments are shown in Figure 1 (b) along with an example track.
The state of the car is a vector ( x, y, v, θ ) where ( x, y ) is its position on the track relative to the current segment, v is its current speed and θ is the heading angle.
An action is a pair ( a, ω ) ∈ Rwhere a is the throttle input and ω is the steering angle.
In this environment, completing each segment is considered a subtask and a task corresponds to completing a sequence of segments— e.g., straight → right → left → sharp-right .
Upon completion of a subtask, the car enters the next segment and a change-of-coordinates is applied to the car’s state which is now relative to the new segment.
The goal here is to learn one option for each subtask such that the agent can perform any task using these options.
If one trains the options independently with the only goal of reaching the end of each segment (e.g., using distance-based rewards), it might (and does) happen that the car reaches the end of a segment in a state that was not part of the initial states used to train the policy corresponding to the next subtask.
Therefore, one should make sure that the initial state distribution used during training includes such states as well—either manually or using dataset aggregation [38].
Furthermore, it is possible that the car reaches a state in the exit region of a segment from which it is challenging to complete the next subtask—e.g., a state in which the car is close to and facing towards a wall.
Our algorithm identifies during training that, in order to perform future subtasks, it is better to reach the end of a segment in a configuration where the car is facing straight relative to the next segment.
As demonstrated in our experiments, this leads to robust options and improved sample efficiency.
Related work.
The options framework [41] is commonly used to model subtask policies as temporally extended actions.
In hierarchical RL [32, 31, 22, 9, 5, 43], options are trained along with a high-level controller that chooses the sequence of options to execute in order to complete a specific high-level task.
There is also work on discovering options—e.g., using intrinsic motivation [30], entropy maximization [10], semi-supervised RL [12], skill chaining [20], expectation maximization [8] and subgoal identification [40].
There has also been a lot of research on planning using learned options [1, 18, 37, 42, 21].
There has been some work on RL for zero-shot generalization [44, 33, 39, 23, 4]; however, in prior work, the learning objective is to maximize average performance with respect to a fixed distribution over tasks as opposed to the worst-case.
Some hierarchical RL algorithms have also been shown to enable few-shot generalization [18] to unseen tasks.
Most closely related to our work is the work on compositional RL in the multi-task setting [17] in which the subtask policies are trained using standard RL algorithms in a naive way without guarantees regarding worst-case performance.
There has also been work on skill composition using transition policies [25]; this method assumes that the subtask policies are fixed and learns one transition policy per subtask which takes the system from an end state to a “favourable” initial state for the subtask.
However, poorly trained subtask policies can lead to situations in which it is not possible to achieve such transitions.
In contrast, our approach trains subtask policies which compose well without requiring additional transition policies.
A recent paper [24] proposes a framework for training subtask policies with the aim of composing them to perform a complex long-horizon task.
However, their approach assumes that the high-level task is fixed and the options are trained to maximize the performance with respect to a specific task.
There has been a lot of research on multi-agent RL algorithms [29, 15, 16, 28, 35, 36, 2] including algorithms for two-agent zero-sum games [6, 45, 27].
In this paper, we utilize the specific structure of our game to obtain a simple algorithm that neither requires solving matrix games nor trains a separate policy for the adversary.
Furthermore, we show that we can obtain an asynchronous RL algorithm which enables learning options in parallel.
Problem Formulation
A multi-task Markov decision process (MDP) is a tuple M =
( S, A, P, Σ , R, F, T, γ, η, σ 0 ) , where
S are the states, A are the actions, P ( s ′ | s, a ) ∈ [0 , 1] is the probability of transitioning from s to s ′ on action a , η is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
Furthermore, Σ is a set of subtasks and for each subtask σ ∈ Σ , R σ : S ×
A → R is a reward function 1 , F σ ⊆ S is a set of final states where the subtask is considered completed and T σ : F σ × S →
[0 , 1] is the jump probability function; upon reaching a state s in F σ the system jumps to a new state s ′ with probability T σ ( s ′ | s ) .
For the sake of clarity, we assume 2 that T σ ( s ′ | s ) =
for any s ′ with s ′ ∈ F σ ′ for some σ ′ .
Finally, σ 0 ∈ Σ is the initial subtask which has to be completed first 3 .
A multi-task MDP can be viewed as a discrete time variant of a hybrid automaton model [17].
In the case of our motivating example, the set of subtasks is given by Σ = { left , right , straight , sharp-left , sharp-right } with F σ denoting the exit region of the segment corresponding to subtask σ .
We use the jump transitions T to model the change-of-coordinates performed upon reaching an exit region.
The reward function R σ for a subtask σ is given by R σ ( s, a, s ′ )
=
−∥ s ′ − c σ ∥ 22 + B · 1 ( s ′ ∈ F σ ) where c σ is the center of the exit region and the subtask completion bonus B is a positive constant.
A task τ is defined to be an infinite sequence 4 of subtasks
τ = σ 0 σ 1 . . .
, and T denotes the set of all tasks.
For any task τ ∈ T , τ [ i ] denotes the i th subtask σ i in τ .
In our setting, the task is chosen by the environment nondeterministically.
Given a task τ , a configuration of the environment is a pair ( s, i ) ∈ S
×
Z ≥ 0 with s / ∈ F
τ
[ i ] denoting that the system is in state s and the current subtask is τ [ i ] .
The initial distribution over configurations ∆ : S × Z ≥
→ [0 , 1] is given by ∆( s, i ) =
η τ [0] ( s ) if i = 0 and 0 otherwise.
The probability of transitioning from ( s, i ) to ( s ′ , j ) on an action a is
Pr(( s ′ , j ) | ( s, i ) , a ) =  
[Equation]
Intuitively, the system transitions to the next subtask if the current subtask is completed and stays in the current subtask otherwise.
A (deterministic) policy is a function π : S → A , where a = π ( s ) is the action to take in state s .
Our goal is to learn one policy π σ for each subtask σ such that the discounted reward over the worst-case task τ is maximized.
Formally, given a set of policies Π = { π σ | σ ∈ Σ } and a task τ , we can define a Markov chain over configurations with initial distribution ∆ and transition probabilities given by P Π (( s ′ , j ) | ( s, i )) =
Pr(( s, j ′ )
| ( s, i ) , π τ [ i ] ( s )) .
We denote by D Π τ the distribution over infinite sequences of configurations ρ
= ( s 0 , i 0 )( s 1 , i 1 ) . . .
generated by τ and Π .
Then, we define the objective function as
[Equation]
These definitions can be naturally extended to stochastic policies as well.
In our motivating example, choosing a large enough completion bonus B guarantees the discounted reward to be higher for runs in which more subtasks are completed.
Our aim is to compute a set of policies Π ∗ ∈ arg max Π J (Π) .
Each subtask policy π σ defines an option [41] o σ = ( π σ , I σ , β σ ) where
I σ = S \ F σ and β σ ( s )
=
1 ( s ∈ F σ ) .
Here, the choice of which option to trigger is made by the environment rather than the agent.
Reduction to Stagewise Markov Games
The problem statement naturally leads to a game theoretic view in which the environment is the adversary.
We can formally reduce the problem to a two-agent zero-sum Markov game
G = ( ¯ S, A 1 , A 2 , ¯ P, ¯ R, ¯ γ, ¯ η ) where ¯ S = S × Σ is the set of states, A
=
A are the actions of agent 1 (the agent learning the options) and A 2 = Σ are the actions of agent 2 (the adversary).
The transition probability function ¯ P : ¯ S ×
A 1 × A 2 × ¯ S → [0 , 1] is given by
[Equation]
| s ) if s ∈ F σ & σ ′ = a otherwise.
We observe that the states are partitioned into two sets ¯ S = S 1 ∪ S 2 where S
= { ( s, σ ) | s / ∈ F σ } is the set of states where agent 1 acts (causing a step in M ) and S
= { ( s, σ ) | s ∈ F σ } is the set of states where agent 2 takes actions (causing a change of subtask); this makes G a stagewise game.
The reward function ¯ R : ¯ S ×
A 1 → R is given by ¯ R (( s, σ ) , a ) =
R σ ( s, a ) if s / ∈ F σ and 0 otherwise.
The discount factor depends on the state and is given by ¯ γ ( s, σ )
= γ if s / ∈ F σ and 1 otherwise; this is because a change of subtask does not invoke a step in M .
The initial state distribution ¯ η is given by ¯ η ( s, σ ) =
η ( s ) 1 ( σ = σ 0 ) .
A run of the game is a sequence ¯ ρ = ¯ s 0 a 10 a 20 ¯ s 1 a 11 a 21 . . .
where ¯ s t ∈ ¯ S and a it ∈
A i .
A (deterministic) policy for agent i is a function
π i : ¯ S
→ A i .
Given policies π 1 and π 2 for agentsand 2, respectively and a state ¯ s ∈ ¯ S we denote by D G ¯ s ( π 1 , π 2 ) the distribution over runs generated by π 1 and π 2 starting at ¯ s .
Then, the value of a state ¯ s is defined by
[Equation]
We are interested in computing a policy π ∗ 1 maximizing
[Equation]
Given a policy π 1 for agent 1, we can construct a policy π σ for any subtask σ given by π σ ( s ) =
π 1 ( s, σ ) ; we denote by Π( π 1 ) the set of subtask policies constructed this way.
The following theorem connects the objective of the game with our multi-task learning objective; all proofs are in
Appendix A.
Theorem 3.1.
For any policy π 1 for agent 1 in G , we have J (Π( π 1 )) ≥ J G ( π 1 ) .
Therefore, J G ( π 1 ) is a lower bound on the objective J (Π( π 1 )) which we seek to maximize.
Now, let us define the optimal value of a state ¯ s by V ∗ (¯ s ) =
max π 1 min π 2 V π 1 ,π 2 (¯ s ) .
The following theorem shows that it is possible to construct a policy π ∗ 1 that maximizes J G ( π 1 ) from the optimal value function V ∗ .
Theorem 3.2.
For any policy π ∗ 1 such that for all ( s, σ ) ∈ S 1 ,
[Equation]
we have that π ∗ 1 ∈ arg max π 1 J G ( π 1 ) .
1: function A SYNC V ALUE I TERATION
( M , V ) 2: while stopping criterion is met do 3: for σ ∈ Σ do // in parallel 4:
Compute W σ ( V ) 5:
V ← F async ( V ) // using Equation
3.1 Value Iteration
In this section, we briefly look at two value iteration algorithms to compute V ∗ which we later adapt in Section 4 to obtain learning algorithms.
Let V
=
{ V : S 1 → R } denote the set of all value functions over S 1 .
Given a value function V ∈ V we define its extension to all of ¯ S using
[Equation]
otherwise.
[Equation]
For a state s ∈ F σ , (cid:74) V (cid:75) ( s, σ ) denotes the worst-case value (according to V ) with respect to the possible choices of next subtask σ ′ .
Now, we consider the Bellman operator F : V → V defined by
[Equation]
for all ( s, σ ) ∈ S 1 .
Let us denote by V ∗ ↓ S 1 the restriction of V ∗ to S 1 .
The following lemma follows straightforwardly giving us our first value iteration procedure.
Theorem 3.3.
F is a contraction mapping with respect to the ℓ ∞ -norm on V and V ∗ ↓ S 1 is the unique fixed point of F with lim n →∞ F n ( V ) =
V ∗ ↓ S 1 for all V ∈ V .
Next we consider an asynchronous value iteration procedure which allows us to parallelize computing subtask policies for different subtasks.
Given a subtask σ and a value function V ∈ V , we define a subtask MDP M Vσ which behaves like M until reaching a final state s ∈ F σ after which it transitions to a dead state ⊥ achieving a reward of (cid:74) V (cid:75) ( s, σ ) .
Formally, M Vσ =
( S σ , A, P σ , R Vσ , γ ) where S σ = S ⊔ {⊥} with ⊥ being a special dead state, P σ ( s ′ | s, a ) =
P ( s ′ | s, a ) if ⊥ ̸ = s / ∈ F σ and P σ ( s ′ | s, a ) = 1 ( s ′ = ⊥ ) otherwise.
The reward function is given by R Vσ ( s, a ) =
R σ ( s, a ) if ⊥ ̸ = s / ∈ F σ , R Vσ ( s, a ) = (cid:74) V (cid:75) ( s, σ ) if ⊥ ̸ = s ∈ F σ and is 0 otherwise.
We denote by W σ ( V ) the optimal value function of the MDP M Vσ .
We then define the asynchronous value iteration operator
F async : V → V using
[Equation]
We can show that repeated application of F async leads to the optimal value function V ∗ of the G .
Theorem 3.4.
For any V ∈ V , lim n →∞ F n async ( V ) → V ∗ ↓ S 1 .
Since each W σ ( V ) can be computed independently, we can parallelize the computation of F async giving us the algorithm in Algorithm 1.
We can also show that it is not necessary to compute W σ ( V ) exactly.
Let V σ = { ¯ V : S σ → R } be the set of all value functions over S σ .
For a fixed V ∈ V , let
F σ,V : V σ → V σ denote the usual Bellman operator for the MDP M Vσ given by
[Equation]
for all ¯ V ∈ V σ and s ∈ S σ .
For any V ∈ V and σ ∈ Σ , we define a corresponding V σ ∈ V σ using
V σ ( s ) =
(cid:74) V (cid:75) ( s, σ ) if s ∈ S and V σ ( ⊥ ) =
Then, for any integer m > 0 and V ∈ V , we can use
F mσ,V ( V σ ) as an approximation to W σ ( V ) .
Let us define F m : V → V using
[Equation]
Intuitively, F m corresponds to performing m steps of value iteration in each subtask MDP M Vσ (which can be parallelized) starting at V σ .
The following theorem guarantees convergence when using F m instead of F async .
Theorem 3.5.
For any V ∈ V and m > 0 , lim n →∞ F nm ( V ) → V ∗ ↓ S 1 .
Inputs: Learning rates α ψ , α θ , entropy weight β and Polyak rate δ .
1: function R OSAC ( α ψ , α θ , β , δ ) 2:
Initialize parameters { ψ σ } σ ∈ Σ , { ψ targ σ } σ ∈ Σ and { θ σ } σ ∈ Σ 3:
Initialize replay buffer B 4: for each iteration do 5: for each episode do 6: s 0 ∼ η 7: σ 0 ← InitialSubtask 8: for each step t do 9: a t ∼ π θ σt ( · | s t ) and s t +1 ∼ P ( · | s, a ) 10:
B ← B ∪ { ( s t , a t , s t +1 ) } 11: if s t +1 ∈ F σ t then 12: s t +1 ∼ T σ t ( · | s t +1 ) 13: σ t +1 ← Greedy ( ε, arg min σ ˜ V
( s t +1 , σ ) , Σ) 14: else 15: σ t +1 ← σ t 16: for each gradient step do 17:
Sample batch B ∼ B 18: for σ ∈ Σ do 19: ψ σ ← ψ σ − α ψ ∇ ψ σ L Q ( ψ σ , B ) 20: θ σ ← θ σ − α θ ∇ θ σ L π ( θ σ , B ) 21: ψ targ σ ← δψ σ + (1 − δ )
ψ targ σ
Learning Algorithms
In this section, we present RL algorithms for solving the game G .
We first consider the finite MDP setting for which we can obtain a modified Q -learning algorithm with a convergence guarantee.
We then present two algorithms based on Soft Actor Critic (S AC ) for the continuous state setting.
4.1 Finite MDP
Assuming finite states and actions, we can obtain a Q -learning variant for solving G which we call
Robust Option Q -learning .
We assume that jump transitions T are known to the learner; this is usually the case since jump transitions are used to model subtask transitions and change-of-coordinates within the controller.
However, we believe that the algorithm can be easily extended to the scenario where T is unknown.
We maintain a function Q : S 1 ×
A → R with Q ( s, σ, a ) denoting Q (( s, σ ) , a ) .
The corresponding value function V Q is defined using V Q ( s, σ ) = max a ∈ A Q ( s, σ, a ) and is extended to all of ¯ S as (cid:74) V Q (cid:75) .
Note that, given a Q -function, the extended value function (cid:74) V Q (cid:75) can be com- puted exactly.
Robust Option Q -learning is an iterative process—in each iteration t , it takes a step (( s, σ ) , a 1 , a 2 , ( s ′ , σ )) in G with ( s, σ ) ∈ S 1 and updates the Q -function using
[Equation]
4) where Q t is the Q -function in iteration t and (cid:74) V Q t (cid:75) is the corresponding extended value function.
Under standard assumptions on the learning rates α t , similar to Q -learning, we can show that Robust
Option Q -learning converges to the optimal Q -function almost surely.
Here, the optimal Q -function is defined by Q ∗ ( s, σ, a ) =
¯ R (( s, σ ) , a ) + γ (cid:80) s ′ ∈ S P ( s ′ | s, a ) V ∗ ( s ′ , σ ) for all ( s, σ ) ∈ S 1 .
Let α t ( s, σ, a ) denote the learning rate used in iteration t if Q t ( s, σ, a ) is updated and 0 otherwise.
Then, we have the following theorem.
Theorem 4.1.
If (cid:80) t α t ( s, σ, a ) = ∞ and (cid:80) t α 2 t ( s, σ, a ) < ∞ for all ( s, σ )
4.2 Continuous States and Actions
In the case of continuous states and actions, we can adapt any Q -function based RL algorithm such as Deep Deterministic Policy Gradients (D DPG ) [26] or Soft Actor Critic (S AC )
[14] to our setting.
Here we present an S AC based algorithm that we call Robust Option S AC (R OSAC ) which is outlined in Algorithm 2.
This algorithm, like S AC , adds an entropy bonus to the reward function to improve exploration.
We maintain two Q -functions for each subtask σ , Q ψ σ : S → R parameterized by ψ σ and a target function Q ψ targ σ parameterized by ψ targ σ .
We also maintain a stochastic subtask policy π θ σ : S →
D ( A ) for each subtask σ where D ( A ) denotes the set of distributions over A .
Given a step ( s, a, s ′ ) in M and a subtask σ with s / ∈ F σ , we define the target value by
[Equation]
where the value (cid:74) V (cid:75) ( s ′ , σ ) is estimated using ˜ V ( s ′ , σ )
=
Q ψ targ σ ( s ′ , ˜ a ) − β log π θ σ (˜ a | s ′ ) with ˜ a ∼ π θ σ ( · | s ′ )
if s ′ / ∈ F σ .
If s ′ ∈ F σ , we estimate (cid:74) V (cid:75) ( s ′ , σ ) using ˜ V
( s ′ , σ ) =
min σ ′ ∈ Σ
˜ V ( s ′′ , σ ′ ) where ˜ V ( s ′′ , σ ′ )
=
Q ψ targ σ ′ ( s ′′ , ˜ a ) − β log π θ σ ′ (˜ a | s ′′ ) with ˜ a ∼ π θ σ ′ ( · | s ′′ ) and s ′′ ∼ T σ ( · | s ′ ) .
Now, given a batch B = { ( s, a, s ′ ) } of steps in M we update ψ σ using one step of gradient descent corresponding to the loss L
[Equation]
and the subtask policy π θ σ is updated using the loss
[Equation]
The gradient ∇ θ σ L π ( θ σ , B ) can be estimated using the reparametrization trick if π θ σ ( · | s ) is a
Gaussian distribution whose parameters are differentiable w.r.t.
θ σ .
We use Polyak averaging to update the target Q -networks { Q ψ targ σ
[Equation]
Note that we do not train a separate policy for the adversary.
During exploration, we use the εgreedy strategy to select subtasks.
We first estimate the “worst” subtask for a state s using ˜ σ = arg min σ ˜ V ( s, σ ) where ˜ V ( s, σ ) is estimated as before.
Then the function Greedy ( ε, ˜ σ, Σ) chooses ˜ σ with probability 1 − ε and picks a subtask uniformly at random from Σ with probability ε .
Asynchronous R OSAC .
We can also obtain an asynchronous version of the above algorithm which lets us train subtask policies in parallel.
Asynchronous Robust Option S AC (A ROSAC ) is outlined in
Algorithm 3.
Here we use one replay buffer for each subtask.
We maintain an initial state distribution ˜ η over S to be used for training every subtask policy { π σ } σ ∈ Σ .
˜ η is represented using a finite set of states D from which a state is sampled uniformly at random.
The value function ˜
V : S × Σ → R is estimated as before.
To be specific, in each iteration, an estimate of any value ˜ V ( s, σ ) is obtained on the fly using the Q -functions and the subtask policies from the previous iteration.
The S AC subroutine runs the standard Soft Actor Critic algorithm for N iterations on the subtask
MDP M ˜ Vσ (defined in Section 3) 5 with initial state distribution
˜ η (defaults to η if D = ∅ ).
It returns the updated parameters along with states X σ visited during exploration with X σ ⊆ F σ .
The states in X σ are used to update the initial state distribution for the next iteration following the Dataset
Aggregation principle [38].
Experiments
We evaluate our algorithms R OSAC and A ROSAC on two multi-task environments; a rooms environment and an F1/10th racing car environment [11].
Inputs: Learning rates α , entropy weight β , Polyak rate δ and number of S AC iterations N .
1: function A ROSAC ( α , β , δ , N ) 2:
Initialize parameters Ψ
= { ψ σ } σ ∈ Σ , Ψ targ = { ψ targ σ } σ ∈ Σ and Θ = { θ σ } σ ∈ Σ 3:
Initialize replay buffers {B σ } σ ∈ Σ and Initialize D = {} 4: for each iteration do 5:
˜ V ← O BTAIN V ALUE E STIMATOR
(Ψ , Θ) 6: for σ ∈ Σ do // in parallel 7: ψ σ , ψ targ σ , θ σ , X σ ← S AC ( M ˜ Vσ , D, ψ σ , ψ targ σ , θ σ , α, β, δ, N ) 8: for σ ∈ Σ do 9: for s ∈ X σ do 10: s ′ ∼ T σ ( · | s ) and D ← D ∪ { s ′ }
Rooms environment.
We consider the environment shown in Figure 2 which depicts a room with walls and exits.
Initially the robot is placed in the green triangle.
The L-shaped obstacles indicate walls within the room that the robot cannot cross.
A state of the system is a position ( x, y ) ∈
R 2 and an action is a pair ( v, θ ) where v is the speed and θ is the heading angle to follow during the next time-step.
There are three exits: left (blue), right (yellow) and up (grey) reaching each of which is a subtask.
Upon reaching an exit, the robot enters another identical room where the exit is identified (via change-of-coordinates) with the bottom entry region of the current room.
A task is a sequence of directions—e.g., left → right → up → right indicating that the robot should reach the left exit followed by the right exit in the subsequent room and so on.
Although the dynamics are simple, the obstacles make learning challenging in the adversarial setting.
F1/10th environment.
This is the environment in the motivating example.
A publicly available simulator [11] of the F1/10th car is used for training and testing.
The policies use the LiDAR measurements from the car as input (as opposed to the state) and we assume that the controller can detect the completion of each segment; as shown in prior work [17], one can train a separate neural network to predict subtask completion.
Baselines.
We compare our approach to three baselines.
The baseline N AIVE trains one controller for each subtask with the only aim of completing the subtask, similar to [17], using a manually designed initial state distribution.
D AGGER is a similar approach which, instead of using a manually designed initial state distribution for training, infers the initial state distribution using the Dataset
Aggregation principle [38].
The M ADDPG baseline solves the game G using the multi-agent RL algorithm proposed in [29] for solving concurrent Markov games with continuous states and actions.
Evaluation.
We evaluate the performance of these algorithms against two adversaries.
One adversary is the random adversary which picks the next subtask uniformly at random from the set of all subtasks.
The other adversary estimates the worst sequence of subtasks for a given set of options using Monte Carlo Tree Search (M CTS ) [19].
The M CTS adversary is trained by assigning a reward of 1 if it selects a subtask which the corresponding policy is unable to complete within a fixed time budget and a reward of 0 otherwise.
For the Rooms environment, we consider subtask sequences of length atmost 5 whereas for the F1/10th environment, we consider sequences of subtasks of length at most 20.
We evaluate both the average number of subtasks completed as well as the probability of completing the set maximum number of subtasks.
Results.
The plots for the rooms environment are shown in Figure 3 and plots for the F1/10th environment are shown in Figure 4.
We can observe that R OSAC is able outperform other approaches and learn robust options.
In the rooms environment, A ROSAC achieves similar performace albeit requiring more samples; however, it has the added benefit of being parallelizable.
In the F1/10th environment, it performs similar to the other baselines.
D AGGER and N AIVE baselines are unable to learn policies that can be used to perform long sequences of subtasks; this is mostly due to the fact
[Figure] [Figure]
that they learn options that cause the system to reach states from which future subtasks are difficult to perform—e.g., in the rooms environment, the agent sometimes reaches the left half of the exits from where it is difficult to reach the right exit in the subsequent room.
Although M ADDPG uses the same reduction to two-player games as R OSAC , it ignores all the structure in the game and treats it as a generic Markov game.
As a result, it learns a separate NN policy for each player which leads to the issue of unstable training, primarily due to the non-stationary nature of the environment observed by either agent.
As shown in the plots, this leads to poor performance when applied to the problem of learning robust options.
Conclusions
We have proposed a framework for training robust options which can be used to perform arbitrary sequences of subtasks.
In our framework, we first reduce the problem to a two-agent zero-sum stagewise Markov game which has a unique structure.
We utilized this structure to design two algorithms, namely R OSAC and A ROSAC , and demonstrated that they outperform existing approaches for training options with respect to multi-task performance.
One potential limitation of our approach is that the set of subtasks is fixed and has to be provided by the user.
An interesting direction for future work is to address this limitation by combining our approach with option discovery methods.
Societal impacts.
Our work seeks to improve reinforcement learning for complex long-horizon tasks.
Any progress in this direction would enable robotics applications both with positive impact— e.g., flexible and general-purpose manufacturing robotics, robots for achieving agricultural tasks, and robots that can be used to perform household chores—and with negative or controversial impact—e.g., military applications.
These issues are inherent in all work seeking to improve the abilities of robots.