Robust Option Learning for Adversarial Generalization
Anonymous Author(s)
AffiliationAddress email
Abstract
Compositional reinforcement learning is a promising approach for training policies to perform complex long-horizon tasks.
Typically, a high-level task is decomposed into a sequence of subtasks and a separate policy is trained to perform eachsubtask.
In this paper, we focus on the problem of training subtask policies in away that they can be used to perform any task; here, a task is given by a sequenceof subtasks.
We aim to maximize the worst-case performance over all tasks asopposed to the average-case performance.
We formulate the problem as a twoagent zero-sum game in which the adversary picks the sequence of subtasks.
Wepropose two RL algorithms to solve this game: one is an adaptation of existingmulti-agent RL algorithms to our setting and the other is an asynchronous versionwhich enables parallel training of subtask policies.
We evaluate our approach ontwo multi-task environments with continuous states and actions and demonstratethat our algorithms outperform state-of-the-art baselines.
Introduction
Reinforcement learning (RL) has proven to be a promising strategy for solving complex controltasks such as walking [13], autonomous driving [17], and dexterous manipulation [3].
However, akey challenge facing the deployment of reinforcement learning in real-world tasks is its high samplecomplexity—to solve any new task requires a training a new policy designed to solve that task.
Onepromising solution is compositional reinforcement learning , where individual options (or skills ) arefirst trained to solve simple tasks; then, these options can be composed together to solve morecomplex tasks [25, 24, 17].
For example, if a driving robot learns how to make left and right turnsand to drive in a straight line, it can then drive along any route composed of these primitives.
A key challenge facing compositional reinforcement learning is the generalizability of the learnedoptions.
In particular, options trained under one distribution of tasks may no longer work well if usedin a new task, since the distribution of initial states from which the options are used may shift.
Analternate approach is to train the options separately to perform specific subtasks, but options trainedthis way might cause the system to reach states from which future subtasks are hard to perform.
Onecan overcome this issue by handcrafting rewards to encourage avoiding such states [17], in whichcase they generalize well, but this approach relies heavily on human time and expertise.
We propose a novel framework that addresses this challenges by formulating the option learningproblem as an adversarial reinforcement learning problem.
At a high level, the adversary choosesthe task that minimizes the reward achieved by composing the available options.
Thus, the goal isto learn a set of robust options that perform well across all potential tasks.
Then, we provide twoalgorithms for solving this problem.
The first adapts existing ideas for using reinforcement learningto solve Markov games to our setting.
Then, the second shows how to leverage the compositional
[Figure]
structure of our problem to learn options in parallel at each step of a value iteration procedure; insome cases, by enabling such parallelism, we can reduce the computational cost of learning.
We validate our approach on two benchmarks: (i) a rooms environment where a point mass robotmust navigate any given sequence of rooms, where the sequence is an arbitrary combination ofstraight, left, and right turns, and (ii) a simulated version of the F1/10th car, where a small racing carmust navigate any racetrack composed of several different track segments.
In both, our empiricalresults demonstrate that robust options are critical for performing well on a wide variety of tasks.
In summary, our contributions are: (i) a game theoretic formulation of the compositional reinforcement learning problem, (ii) two algorithms for solving this problem, and (iii) an empirical evaluationdemonstrating the effectiveness of our approach.
Motivating example.
Let us consider a small scale autonomous racing car shown in Figure 1 (a).
We would like to train a controller that can be used to navigate the car through all tracks constructedusing five kinds of segments; the possible segments are shown in Figure 1 (b) along with an exampletrack.
The state of the car is a vector ( x, y, v, θ ) where ( x, y ) is its position on the track relative tothe current segment, v is its current speed and θ is the heading angle.
An action is a pair ( a, ω ) ∈ R 2where a is the throttle input and ω is the steering angle.
In this environment, completing eachsegment is considered a subtask and a task corresponds to completing a sequence of segments—e.g., straight → right → left → sharp-right .
Upon completion of a subtask, the car entersthe next segment and a change-of-coordinates is applied to the car’s state which is now relative to thenew segment.
The goal here is to learn one option for each subtask such that the agent can performany task using these options.
If one trains the options independently with the only goal of reaching the end of each segment(e.g.
, using distance-based rewards), it might (and does) happen that the car reaches the end of asegment in a state that was not part of the initial states used to train the policy corresponding to thenext subtask.
Therefore, one should make sure that the initial state distribution used during trainingincludes such states as well—either manually or using dataset aggregation [38].
Furthermore, it ispossible that the car reaches a state in the exit region of a segment from which it is challenging tocomplete the next subtask—e.g., a state in which the car is close to and facing towards a wall.
Ouralgorithm identifies during training that, in order to perform future subtasks, it is better to reach theend of a segment in a configuration where the car is facing straight relative to the next segment.
Asdemonstrated in our experiments, this leads to robust options and improved sample efficiency.
Related work.
The options framework [41] is commonly used to model subtask policies as temporally extended actions.
In hierarchical RL [32, 31, 22, 9, 5, 43], options are trained along with ahigh-level controller that chooses the sequence of options to execute in order to complete a specifichigh-level task.
There is also work on discovering options—e.g., using intrinsic motivation [30],entropy maximization [10], semi-supervised RL [12], skill chaining [20], expectation maximization [8] and subgoal identification [40].
There has also been a lot of research on planning usinglearned options [1, 18, 37, 42, 21].
There has been some work on RL for zero-shot generalization [44, 33, 39, 23, 4]; however, in priorwork, the learning objective is to maximize average performance with respect to a fixed distributionover tasks as opposed to the worst-case.
Some hierarchical RL algorithms have also been shown toenable few-shot generalization [18] to unseen tasks.
Most closely related to our work is the work on compositional RL in the multi-task setting [17] in which the subtask policies are trained usingstandard RL algorithms in a naive way without guarantees regarding worst-case performance.
There has also been work on skill composition using transition policies [25]; this method assumesthat the subtask policies are fixed and learns one transition policy per subtask which takes the systemfrom an end state to a “favourable” initial state for the subtask.
However, poorly trained subtaskpolicies can lead to situations in which it is not possible to achieve such transitions.
In contrast, ourapproach trains subtask policies which compose well without requiring additional transition policies.
A recent paper [24] proposes a framework for training subtask policies with the aim of composingthem to perform a complex long-horizon task.
However, their approach assumes that the high-leveltask is fixed and the options are trained to maximize the performance with respect to a specific task.
There has been a lot of research on multi-agent RL algorithms [29, 15, 16, 28, 35, 36, 2] includingalgorithms for two-agent zero-sum games [6, 45, 27].
In this paper, we utilize the specific structureof our game to obtain a simple algorithm that neither requires solving matrix games nor trains aseparate policy for the adversary.
Furthermore, we show that we can obtain an asynchronous RLalgorithm which enables learning options in parallel.
Problem Formulation
A multi-task Markov decision process (MDP) is a tuple M =
( S, A, P, Σ , R, F, T, γ, η, σ 0 ) , where
S are the states, A are the actions, P ( s ′ | s, a ) ∈ [0 , 1] is the probability of transitioning from s tos ′ on action a , η is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
Furthermore,Σ is a set of subtasks and for each subtask σ ∈ Σ , R σ : S ×
A → R is a reward function 1 , F σ ⊆
Sis a set of final states where the subtask is considered completed and T σ : F σ × S →
[0 , 1] is thejump probability function; upon reaching a state s in F σ the system jumps to a new state s ′ withprobability T σ ( s ′ | s ) .
For the sake of clarity, we assume 2 that T σ ( s ′ | s ) =
for any s ′ withs ′ ∈ F σ ′ for some σ ′ .
Finally, σ 0 ∈ Σ is the initial subtask which has to be completed first 3 .
Amulti-task MDP can be viewed as a discrete time variant of a hybrid automaton model [17].
In the case of our motivating example, the set of subtasks is given byΣ = { left , straight , sharp sharp
} with F σ denoting the exit region of the segment corresponding to subtask σ .
We use the jumptransitions T to model the change-of-coordinates performed upon reaching an exit region.
Thereward function R σ for a subtask σ is given by R σ ( s, a, s ′ )
=
−∥ s ′ − c σ ∥ 22 + B · 1 ( s ′ ∈ F σ ) wherec σ is the center of the exit region and the subtask completion bonus B is a positive constant.
A task τ is defined to be an infinite sequence 4 of subtasks
τ = σ 0 σ 1 . . .
, and T denotes the set of all tasks.
For any task τ ∈ T , τ [ i ] denotes the i th subtask σ i in τ .
In our setting, the task is chosen by the environment nondeterministically.
Given a task τ , a configuration of the environment is a pair ( s, i ) ∈ S
×
Z ≥ 0 with s / ∈ F
τ
[ i ] denoting that the system is in state s and the current subtask is τ [ i ] .
The initial distribution over configurations ∆ : S × Z ≥
→ [0 , 1] is given by ∆( s, i ) =
η τ [0] ( s ) if i = 0 and 0 otherwise.
The probability of transitioning from ( s, i ) to ( s ′ , j ) on an action a is
Pr(( s ′ , j ) | ( s, i ) , a ) = 
 (cid:80) s ′′ ∈ F τ [ i ] P ( s ′′ | s, a ) T τ [ i ] ( s ′ | s ′′ ) if j = i
+
P ( s ′ | s, a )
if j = i otherwise.
Intuitively, the system transitions to the next subtask if the current subtask is completed and stays in the current subtask otherwise.
A (deterministic) policy is a function π : S → A , where a = π ( s ) is the action to take in state s .
Our goal is to learn one policy π σ for each subtask σ such that the discounted reward over the worst-case task τ is maximized.
Formally, given a set of policies Π = { π σ | σ ∈ Σ } and a task τ , we can define a Markov chain over configurations with initial distribution ∆ and transition probabilities given by P Π (( s ′ , j ) | ( s, i )) =
Pr(( s, j ′ )
| ( s, i ) , π τ [ i ] ( s )) .
We denote by D Π τ the distribution over infinite sequences of configurations ρ
= ( s 0 , i 0 )( s 1 , i 1 ) . . .
generated by τ and Π .
Then, we define the objective function as
[Equation]
These definitions can be naturally extended to stochastic policies as well.
In our motivating example, choosing a large enough completion bonus B guarantees the discounted reward to behigher for runs in which more subtasks are completed.
Our aim is to compute a set of policiesΠ ∗ ∈ arg max Π J (Π) .
Each subtask policy π σ defines an option [41] o σ = ( π σ , I σ , β σ ) where
I σ = S \ F σ and β σ ( s )
=
1 ( s ∈ F σ ) .
Here, the choice of which option to trigger is made by theenvironment rather than the agent.
Reduction to Stagewise Markov Games
The problem statement naturally leads to a game theoretic view in which the environment is the adversary.
We can formally reduce the problem to a two-agent zero-sum Markov game
G = ( ¯ S, A 1 , A 2 , ¯ P, ¯ R, ¯ γ, ¯ η ) where ¯ S = S × Σ is the set of states, A {+
+} =
A are the actions of agent 1 (the agent learning the options) and A 2 = Σ are the actions of agent 2 (the adversary).
The transition probability function ¯ P : ¯ S × A [Equation] ×
A
[Equation] × [Equation] S → [0 , 1] is given by
[Equation]
We observe that the states are partitioned into two sets ¯ S = S 1 ∪ S 2 where S
= { ( s, σ ) | s / ∈ F σ }is the set of states where agent 1 acts (causing a step in M ) and S
= { ( s, σ ) | s ∈ F σ } is the setof states where agent 2 takes actions (causing a change of subtask); this makes G a stagewise game.
The reward function ¯ R : ¯ S ×
A 1 → R is given by ¯ R (( s, σ ) , a ) =
R σ ( s, a ) if s / ∈ F σ and 0 otherwise.
The discount factor depends on the state and is given by ¯ γ ( s, σ )
= γ if s / ∈ F σ and 1 otherwise; thisis because a change of subtask does not invoke a step in M .
The initial state distribution ¯ η is givenby
¯ η
( s, σ )
=
η ( s ) 1 ( σ = σ 0 ) .
A run of the game is a sequence ¯ ρ = ¯ s 0 a 10 a 20 ¯ s 1 a 11 a 21 . . .
where ¯ s t ∈
¯
Sand a it ∈
A i .
A (deterministic) policy for agent i is a function
π i : ¯ S
→ A i .
Given policies π 1 and π 2 for agentsand 2, respectively and a state ¯ s ∈ ¯ S we denote by D G ¯ s ( π 1 , π 2 ) the distribution over runs generated by π 1 and π 2 starting at ¯ s .
Then, the value of a state ¯ s is defined by
[Equation]
We are interested in computing a policy π ∗ 1 maximizing
[Equation]
Given a policy π 1 for agent 1, we can construct a policy π σ for any subtask σ given by π σ ( s ) =
π 1 ( s, σ ) ; we denote by Π( π 1 ) the set of subtask policies constructed this way.
The following theorem connects the objective of the game with our multi-task learning objective; all proofs are in
Appendix A.
Theorem 3.1.
For any policy π 1 for agent 1 in G , we have J (Π( π 1 )) ≥ J G ( π 1 ) .
Therefore, J G ( π 1 ) is a lower bound on the objective J (Π( π 1 )) which we seek to maximize.
Now,let us define the optimal value of a state ¯ s by V ∗ (¯ s ) =
max π 1 min π 2 V π 1 ,π 2 (¯ s ) .
The followingtheorem shows that it is possible to construct a policy π ∗ 1 that maximizes J G ( π 1 ) from the optimalvalue function V ∗ .
Theorem 3.2.
For any policy π ∗ 1 such that for all ( s, σ ) ∈ S 1 ,
[Equation]
we have that π ∗ 1 ∈ arg max π 1 J G ( π 1 ) .
1: function A SYNC V ALUE I TERATION
( M , V ) 2: while stopping criterion is met do 3: for σ ∈ Σ do // in parallel 4:
Compute W σ ( V ) 5:
V ← F async ( V ) // using Equation
3.1 Value Iteration
In this section, we briefly look at two value iteration algorithms to compute V ∗ which we later adaptin Section 4 to obtain learning algorithms.
Let V
=
{ V : S 1 → R } denote the set of all valuefunctions over S 1 .
Given a value function V ∈ V we define its extension to all of ¯ S using
[Equation]
otherwise.
[Equation] [Equation]
For a state s ∈ F σ , (cid:74) V (cid:75) ( s, σ ) denotes the worst-case value (according to V ) with respect to thepossible choices of next subtask σ ′ .
Now, we consider the Bellman operator F : V → V defined by
[Equation]
[Equation]
for all ( s, σ ) ∈ S 1 .
Let us denote by V ∗ ↓ S 1 the restriction of V ∗ to S 1 .
The following lemmafollows straightforwardly giving us our first value iteration procedure.
Theorem 3.3.
F is a contraction mapping with respect to the ℓ ∞ -norm on V and V ∗ ↓ S 1 is theunique fixed point of F with lim n →∞ F n ( V ) =
V ∗ ↓ S 1 for all V ∈ V .
Next we consider an asynchronous value iteration procedure which allows us to parallelize computing subtask policies for different subtasks.
Given a subtask σ and a value function V ∈ V , wedefine a subtask MDP M Vσ which behaves like M until reaching a final state s ∈ F σ after which ittransitions to a dead state ⊥ achieving a reward of (cid:74) V (cid:75) ( s, σ ) .
Formally, M Vσ =
( S σ , A, P σ , R Vσ , γ )where S σ = S ⊔ {⊥} with ⊥ being a special dead state, P σ ( s ′ | s, a ) =
P ( s ′ | s, a ) if ⊥ ̸ = s / ∈ F σand P σ ( s ′ | s, a )
=
1 ( s ′ = ⊥ )
otherwise.
The reward function is given by R Vσ ( s, a ) =
R σ ( s, a ) if⊥ ̸ = s / ∈ F σ ,
R Vσ ( s, a ) =
(cid:74) V (cid:75) ( s, σ ) if ⊥ ̸ = s ∈ F σ and is 0 otherwise.
We denote by W σ ( V ) theoptimal value function of the MDP M Vσ .
We then define the asynchronous value iteration operator
F async : V → V using
[Equation] [Equation]
We can show that repeated application of F async leads to the optimal value function V ∗ of the G .
Theorem 3.4.
For any V ∈ V , lim n →∞ F n async ( V ) → V ∗ ↓ S 1 .
Since each W σ ( V ) can be computed independently, we can parallelize the computation of F async giving us the algorithm in Algorithm 1.
We can also show that it is not necessary to compute W σ ( V ) exactly.
Let V σ = { ¯ V : S σ → R } be the set of all value functions over S σ .
For a fixed V ∈ V , let
F σ,V : V σ → V σ denote the usual Bellman operator for the MDP M Vσ given by
[Equation]
for all ¯ V ∈ V σ and s ∈ S σ .
For any V ∈ V and σ ∈ Σ , we define a corresponding V σ ∈ V σ using
V σ ( s ) =
(cid:74) V (cid:75) ( s, σ ) if s ∈ S and V σ ( ⊥ ) =
Then, for any integer m > 0 and V ∈ V , we can use
F mσ,V ( V σ ) as an approximation to W σ ( V ) .
Let us define F m : V → V using
[Equation]
Intuitively, F m corresponds to performing m steps of value iteration in each subtask MDP M Vσ(which can be parallelized) starting at V σ .
The following theorem guarantees convergence whenusing F m instead of F async .
Theorem 3.5.
For any V ∈ V and m > 0 , lim n →∞ F nm ( V ) → V ∗ ↓ S 1 .
Inputs: Learning rates α ψ , α θ , entropy weight β and Polyak rate δ .
1: function R OSAC ( α ψ , α θ , β , δ ) 2:
Initialize parameters { ψ σ } σ ∈ Σ , { ψ targ σ } σ ∈ Σ and { θ σ } σ ∈ Σ 3:
Initialize replay buffer B 4: for each iteration do 5: for each episode do 6: s 0 ∼ η 7: σ 0 ← InitialSubtask 8: for each step t do 9: a t ∼ π θ σt ( · | s t ) and s t +1 ∼ P ( · | s, a ) 10:
B ← B ∪ { ( s t , a t , s t +1 ) } 11: if s t +1 ∈ F σ t then 12: s t +1 ∼ T σ t ( · | s t +1 ) 13: σ t +1 ← Greedy ( ε, arg min σ ˜ V
( s t +1 , σ ) , Σ) 14: else 15: σ t +1 ← σ t 16: for each gradient step do 17:
Sample batch B ∼ B 18: for σ ∈ Σ do 19: ψ σ ← ψ σ − α ψ ∇ ψ σ L Q ( ψ σ , B ) 20: θ σ ← θ σ − α θ ∇ θ σ L π ( θ σ , B ) 21: ψ targ σ ← δψ σ + (1 − δ )
ψ targ σ
Learning Algorithms
In this section, we present RL algorithms for solving the game G .
We first consider the finite MDPsetting for which we can obtain a modified Q -learning algorithm with a convergence guarantee.
Wethen present two algorithms based on Soft Actor Critic (S AC ) for the continuous state setting.
4.1 Finite MDP
Assuming finite states and actions, we can obtain a Q -learning variant for solving G which we call
Robust Option Q -learning .
We assume that jump transitions T are known to the learner; this is usually the case since jump transitions are used to model subtask transitions and change-of-coordinateswithin the controller.
However, we believe that the algorithm can be easily extended to the scenariowhere T is unknown.
We maintain a function Q : S 1 ×
A → R with Q ( s, σ, a ) denoting Q (( s, σ ) , a ) .
The corresponding value function V Q is defined using V Q ( s, σ ) = max a ∈ A Q ( s, σ, a ) and is extended toall of ¯ S as (cid:74) V Q (cid:75) .
Note that, given a Q -function, the extended value function (cid:74) V Q (cid:75) can be com- 170 puted exactly.
Robust Option Q -learning is an iterative process—in each iteration t , it takes a step(( s, σ ) , a 1 , a 2 , ( s ′ , σ )) in G with ( s, σ ) ∈ S 1 and updates the Q -function using
[Equation]
[Equation]
where Q t is the Q -function in iteration t and (cid:74) V Q t (cid:75) is the corresponding extended value function.
Under standard assumptions on the learning rates α t , similar to Q -learning, we can show that Robust
Option Q -learning converges to the optimal Q -function almost surely.
Here, the optimal Q -functionis defined by Q ∗ ( s, σ, a ) =
¯ R (( s, σ ) , a ) + γ (cid:80) s ′ ∈ S P ( s ′ | s, a ) V ∗ ( s ′ , σ ) for all ( s, σ ) ∈ S 1 .
Letα t ( s, σ, a ) denote the learning rate used in iteration t if Q t ( s, σ, a ) is updated and 0 otherwise.
Then,we have the following theorem.
Theorem 4.1.
If (cid:80) t α t ( s, σ, a ) = ∞ and (cid:80)
4.2 Continuous States and Actions
In the case of continuous states and actions, we can adapt any Q -function based RL algorithm suchas Deep Deterministic Policy Gradients (D DPG ) [26] or Soft Actor Critic (S AC )
[14] to our setting.
Here we present an S AC based algorithm that we call Robust Option S AC (R OSAC ) which is outlinedin Algorithm 2.
This algorithm, like S AC , adds an entropy bonus to the reward function to improveexploration.
We maintain two Q -functions for each subtask σ , Q ψ σ : S → R parameterized by ψ σ and a target function Q ψ targ σ parameterized by ψ targ σ .
We also maintain a stochastic subtask policy π θ σ : S →
D ( A ) for each subtask σ where D ( A ) denotes the set of distributions over A .
Given a step ( s, a, s ′ ) in M and a subtask σ with s / ∈ F σ , we define the target value by
[Equation]
where the value (cid:74) V (cid:75) ( s ′ , σ ) is estimated using ˜ V ( s ′ , σ )
=
Q ψ targ σ ( s ′ , ˜ a ) − β log π θ σ (˜ a | s ′ ) with ˜ a ∼ π θ σ ( · | s ′ )
if s ′ / ∈ F σ .
If s ′ ∈ F σ , we estimate (cid:74) V (cid:75) ( s ′ , σ ) using ˜ V
( s ′ , σ ) =
min σ ′ ∈ Σ
˜ V ( s ′′ , σ ′ ) where ˜ V ( s ′′ , σ ′ )
=
Q ψ targ σ ′ ( s ′′ , ˜ a ) − β log π θ σ ′ (˜ a | s ′′ ) with ˜ a ∼ π θ σ ′ ( · | s ′′ ) and s ′′ ∼ T σ ( · | s ′ ) .
Now, given a batch B = { ( s, a, s ′ ) } of steps in M we update ψ σ using one step of gradient descent corresponding to the loss
[Equation]
and the subtask policy π θ σ is updated using the loss
[Equation]
The gradient ∇ θ σ L π ( θ σ , B ) can be estimated using the reparametrization trick if π θ σ ( · | s ) is a
Gaussian distribution whose parameters are differentiable w.r.t.
θ σ .
We use Polyak averaging toupdate the target Q -networks { Q ψ targ σ |
[Equation]
∈ Σ } .
Note that we do not train a separate policy for the adversary.
During exploration, we use the εgreedy strategy to select subtasks.
We first estimate the “worst” subtask for a state s using ˜ σ =arg min σ ˜ V ( s, σ ) where ˜ V ( s, σ ) is estimated as before.
Then the function Greedy ( ε, ˜ σ, Σ) chooses˜ σ with probability 1 − ε and picks a subtask uniformly at random from Σ with probability ε .
Asynchronous R OSAC .
We can also obtain an asynchronous version of the above algorithm whichlets us train subtask policies in parallel.
Asynchronous Robust Option S AC (A ROSAC ) is outlined in
Algorithm 3.
Here we use one replay buffer for each subtask.
We maintain an initial state distribution˜ η over S to be used for training every subtask policy { π σ } σ ∈ Σ .
˜ η is represented using a finite set ofstates D from which a state is sampled uniformly at random.
The value function ˜
V : S × Σ → Ris estimated as before.
To be specific, in each iteration, an estimate of any value ˜ V ( s, σ ) is obtainedon the fly using the Q -functions and the subtask policies from the previous iteration.
The S AC subroutine runs the standard Soft Actor Critic algorithm for N iterations on the subtask
MDP M ˜ Vσ (defined in Section 3) 5 with initial state distribution
˜ η (defaults to η if D = ∅ ).
It returnsthe updated parameters along with states X σ visited during exploration with X σ ⊆ F σ .
The statesin X σ are used to update the initial state distribution for the next iteration following the Dataset
Aggregation principle [38].
Experiments
We evaluate our algorithms R OSAC and A ROSAC on two multi-task environments; a rooms environment and an F1/10th racing car environment [11].
Inputs: Learning rates α , entropy weight β , Polyak rate δ and number of S AC iterations N .
1: function A ROSAC ( α , β , δ , N ) 2:
Initialize parameters Ψ
= { ψ σ } σ ∈ Σ , Ψ targ = { ψ targ σ } σ ∈ Σ and Θ = { θ σ } σ ∈ Σ 3:
Initialize replay buffers {B σ } σ ∈ Σ and Initialize D = {} 4: for each iteration do 5:
˜ V ← O BTAIN V ALUE E STIMATOR
(Ψ , Θ) 6: for σ ∈ Σ do // in parallel 7: ψ σ , ψ targ σ , θ σ , X σ ← S AC ( M ˜ Vσ , D, ψ σ , ψ targ σ , θ σ , α, β, δ, N ) 8: for σ ∈ Σ do 9: for s ∈ X σ do 10: s ′ ∼ T σ ( · | s ) and D ← D ∪ { s ′ }
Rooms environment.
We consider the environment shown in Figure 2 which depicts a room with walls and exits.
Initially the robotis placed in the green triangle.
The L-shaped obstacles indicate wallswithin the room that the robot cannot cross.
A state of the system is aposition ( x, y ) ∈
R 2 and an action is a pair ( v, θ ) where v is the speedand θ is the heading angle to follow during the next time-step.
Thereare three exits: left (blue), right (yellow) and up (grey) reaching eachof which is a subtask.
Upon reaching an exit, the robot enters anotheridentical room where the exit is identified (via change-of-coordinates)with the bottom entry region of the current room.
A task is a sequenceof directions—e.g., left → right → up → right indicating thatthe robot should reach the left exit followed by the right exit in thesubsequent room and so on.
Although the dynamics are simple, theobstacles make learning challenging in the adversarial setting.
F1/10th environment.
This is the environment in the motivating example.
A publicly availablesimulator [11] of the F1/10th car is used for training and testing.
The policies use the LiDARmeasurements from the car as input (as opposed to the state) and we assume that the controller candetect the completion of each segment; as shown in prior work [17], one can train a separate neuralnetwork to predict subtask completion.
Baselines.
We compare our approach to three baselines.
The baseline N AIVE trains one controllerfor each subtask with the only aim of completing the subtask, similar to [17], using a manuallydesigned initial state distribution.
D AGGER is a similar approach which, instead of using a manuallydesigned initial state distribution for training, infers the initial state distribution using the Dataset
Aggregation principle [38].
The M ADDPG baseline solves the game G using the multi-agent RLalgorithm proposed in [29] for solving concurrent Markov games with continuous states and actions.
Evaluation.
We evaluate the performance of these algorithms against two adversaries.
One adversary is the random adversary which picks the next subtask uniformly at random from the set of allsubtasks.
The other adversary estimates the worst sequence of subtasks for a given set of optionsusing Monte Carlo Tree Search (M CTS ) [19].
The M CTS adversary is trained by assigning a rewardof 1 if it selects a subtask which the corresponding policy is unable to complete within a fixed timebudget and a reward of 0 otherwise.
For the Rooms environment, we consider subtask sequences oflength atmost 5 whereas for the F1/10th environment, we consider sequences of subtasks of lengthat most 20.
We evaluate both the average number of subtasks completed as well as the probabilityof completing the set maximum number of subtasks.
Results.
The plots for the rooms environment are shown in Figure 3 and plots for the F1/10thenvironment are shown in Figure 4.
We can observe that R OSAC is able outperform other approachesand learn robust options.
In the rooms environment, A ROSAC achieves similar performace albeitrequiring more samples; however, it has the added benefit of being parallelizable.
In the F1/10thenvironment, it performs similar to the other baselines.
D AGGER and N AIVE baselines are unable tolearn policies that can be used to perform long sequences of subtasks; this is mostly due to the fact
[Figure] [Figure]
that they learn options that cause the system to reach states from which future subtasks are difficultto perform—e.g., in the rooms environment, the agent sometimes reaches the left half of the exitsfrom where it is difficult to reach the right exit in the subsequent room.
Although M ADDPG uses thesame reduction to two-player games as R OSAC , it ignores all the structure in the game and treats it asa generic Markov game.
As a result, it learns a separate NN policy for each player which leads to theissue of unstable training, primarily due to the non-stationary nature of the environment observedby either agent.
As shown in the plots, this leads to poor performance when applied to the problemof learning robust options.
Conclusions
We have proposed a framework for training robust options which can be used to perform arbitrarysequences of subtasks.
In our framework, we first reduce the problem to a two-agent zero-sumstagewise Markov game which has a unique structure.
We utilized this structure to design two algorithms, namely R OSAC and A ROSAC , and demonstrated that they outperform existing approachesfor training options with respect to multi-task performance.
One potential limitation of our approachis that the set of subtasks is fixed and has to be provided by the user.
An interesting direction forfuture work is to address this limitation by combining our approach with option discovery methods.
Societal impacts.
Our work seeks to improve reinforcement learning for complex long-horizontasks.
Any progress in this direction would enable robotics applications both with positive impact—e.g., flexible and general-purpose manufacturing robotics, robots for achieving agricultural tasks,and robots that can be used to perform household chores—and with negative or controversialimpact—e.g., military applications.
These issues are inherent in all work seeking to improve theabilities of robots.