P ROGRAMMATIC R EINFORCEMENT L EARNING WITHOUT O RACLES
A BSTRACT
Deep reinforcement learning (RL) has led to encouraging successes in many challenging control tasks.
However, a deep RL model lacks interpretability due to the difﬁculty of identifying how the model’s control logic relates to its network structure.
Programmatic policies structured in more interpretable representations emerge as a promising solution.
Yet two shortcomings remain: First, synthesizing programmatic policies requires optimizing over the discrete and non-differentiable search space of program architectures.
Previous works are suboptimal because they only enumerate program architectures greedily guided by a pretrained RL oracle.
Second, these works do not exploit compositionality, an important programming concept, to reuse and compose primitive functions to form a complex function for new tasks.
Our ﬁrst contribution is a programmatically interpretable RL framework that conducts program architecture search on top of a continuous relaxation of the architecture space deﬁned by programming language grammar rules.
Our algorithm allows policy architectures to be learned with policy parameters via bilevel optimization using efﬁcient policy-gradient methods, and thus does not require a pretrained oracle.
Our second contribution is improving programmatic policies to support compositionality by integrating primitive functions learned to grasp task-agnostic skills as a composite program to solve novel RL problems.
Experiment results demonstrate that our algorithm excels in discovering optimal programmatic policies that are highly interpretable.
I NTRODUCTION
A growing body of research has explored programs in a domain-speciﬁc programming language as a new RL policy representation that intentionally encourages policy interpretability.
Yet, learning a policy as a high-level program in a structured representation is challenging.
This is because algorithms must jointly identify a reasonable program architecture to allow for sufﬁcient expressiveness while optimizing the parameters of the program modules in the architecture.
For example, depending on the shape of a maze, walking a robot to different goals on the maze by a program may require various if-then-else conditions to travel along different paths, the number of which might not be known to the agent before training.
To curb the non-differentiable program architecture search space, existing programmatic policy learning algorithms either learn from a pretrained program embedding space that must support smooth interpolation (Trivedi et al., 2021), or must be guided by the supervision of a pretrained oracle (e.g. a neural network policy trained by RL) via a teacher-student learning paradigm (Bastani et al., 2018; Silver et al., 2020;
Inala et al., 2020; Verma et al., 2018; 2019).
The task of imitating an oracle is signiﬁcantly simpler than the full RL problem.
However, since the policy space of neural networks and that of programmatic policies are very different, a signiﬁcant performance gap exists between an imitating program and its RL oracle (Verma et al., 2018).
The ﬁrst contribution of our paper is a framework to synthesize interpretable and differentiable programmatic policies solely from reward signals by policy gradient methods, without needing any oracles or pretraining .
A conceivable way of synthesizing program architectures would be to enumerate all possible architectures induced by the grammar of a domain-speciﬁc policy language, run standard RL for each to ﬁnd the optimal values of its unknown parameters, and return the best program.
However, doing so is computationally expensive as each RL trial may explore millions of environment steps.
Inspired by recent advances in differentiable neural architecture search e.g.
DARTs (Liu et al., 2019b), we relax the discrete program architecture search space to be continuous.
Speciﬁcally, we encode program architecture synthesis as learning the probability distribution over all possible architecture derivations (up to a certain bound on program abstract syntax tree depth) induced by a policy language grammar.
This enables our RL algorithm to jointly optimize program architectures and the parameters of program modules via policy-gradient methods.
Our second contribution is improving programmatic policies to support compositionality — the integration of primitive functions trained to grasp basic, task-agnostic skills (e.g. running forward or jumping) into a new complex function as a composite model to solve novel RL problems (e.g. jumping over multiple hurdles to reach a target).
As opposed to policy ensemble models based on neural networks (Qureshi et al., 2020), our programmatically composite models interpret how primitive functions are composed under different environment conditions based on an RL agent’s perceptions, and naturally generalize to novel scenarios.
We further apply programmatic policies to address challenging hierarchical RL problems.
Our solution leverages the speciﬁcations of primitive functions to create an optimal high-level control plan via a satisﬁability constraint solver and implements the high-level plans by learned composition of primitive functions.
Finally, we benchmark our method against the state-of-the-art RL methods.
Our results demonstrate that the programmatic RL framework is able to solve extremely hard RL problems using highly interpretable policies with improved task performance.
P ROBLEM M OTIVATION AND F ORMULATION
We study how to express RL policies as differentiable programs, which use symbolic language constructs to compose a set of parameterized primitive modules.
To control an agent, a programmatic policy takes an environment state as input and computes an action as return for the agent to execute.
[Equation]
We view a programmatic policy as a pair ( E, θ ) , where
E is a discrete program architecture and θ is a vector of real-valued parameters of the program.
A program architecture E is structured based on the context-free grammar (Hopcroft et al., 2007) of a policy DSL.
In this paper, we consider the context-free grammar depicted in the standard
Backus-Naur form (Winskel, 1993) in Fig.
A vertical bar “ | ” indicates choice.
Such a grammar consists of a set of production rules X ::= σ 1 σ 2 · · · σ j where X is a nonterminal and σ 1 , · · · , σ j are either terminals or nonterminals.
For example, we may expand the nonteriminal E 1 in a partial program if B 1 then C 1 else Eto if B 1 then C 1 else ( if B 2 then C 2 else E 2 ).
The nonterminals E and B stand for program expressions that evaluate to action values in R m and Booleans, respectively, where m is the action dimension size.
We represent a state input to a programmatic policy as s = { x 1 : ν 1 , x 2 : ν 2 , . . .
, x n } where n is the state dimension size and ν i = s
[ x i ] is the value of x i in s .
As usual, the unbounded variables in X =
[ x 1 , x 2 , . . .
, x n ] are assumed to be input variables (state variables in our context).
A terminal in this grammar is a symbol that can appear in a program’s code, e.g. the if symbol and x i .
The semantics of a program in E is mostly standard and given by a function (cid:74) E (cid:75) ( s ) , deﬁned for each
DSL construct.
For example, (cid:74) x
i (cid:75) ( s ) =
s [ x i ] reads the value of a variable x
i in a state input s .
A policy may use an if-then-else branching construct.
To avoid discontinuities for differentiability, we interpret its semantics in terms of a smooth approximation where σ is the sigmoid function: (cid:74) if B then C else E (cid:75) ( s ) =
σ ( (cid:74)
[Equation]
Thus, any policy programmed in this grammar becomes a differentiable program.
C is a controller used by a programmatic policy.
During execution, the policy can invoke a set of controllers under different environment conditions, according to the activation of B conditions in the program.
We consider three DSLs depending on how C is structured for afﬁne , ensemble , and PID policies.
Afﬁne Policies.
The DSL for afﬁne policies allows C to be expanded as an afﬁne transformation:
[Equation]
if θ 1 c + θ T 1 · X > 0 then ( 95% · π UP ( s ) + 5% · π LEFT ( s ))
←− Branchelse if θ 2 c + θ T 2 · X > 0 then ( 95% · π LEFT ( s ) + 5% · π RIGHT ( s ))
←− Branchelse ( 13% · π DOWN ( s ) + 87% · π RIGHT ( s ))
←− Branch
X =
[ x, y, G x , G y , arctan yx , (cid:107) x, y (cid:107) 2 ] θ
=
[ − 2 . 052 , 0 . 049 , 0 .
241 , 1 . 443 ] , θ 1 c = − 0 .
202 θ 2 =
[ 1 . 333 , 2 . 204 , − 2 .
331 ] , θ 2 c = − 0 .
where θ ∈ R m ·|X| , θ c ∈ R m are policy parameters.
Particularly, C Afﬁne can be as simple as some (learned) constants θ c .
An example afﬁne policy is given in Appendix K.1.
Ensemble Policies.
The most important feature of our programmatic model is compositionality — composing and reusing task-agnostic primitives in new programs to solve novel problems.
The DSL for ensemble policies includes pre-acquired primitives π 1 , · · · , π N as callable library functions:
[Equation]
C π explicitly compose primitive functions (e.g. running forward or jumping) hierarchically into a complex program (e.g. jumping over multiple hurdles to reach a target) where θ 1 , · · · , θ N ∈ Rparameterize a primitive combination.
The input space of a primitive function can be different from that of a program (formally deﬁned below).
The semantics of C π is deﬁned as follows:
[Equation]
q
[Equation]
Here the composition weights { q i } Ni =0 for primitive ensemble are computed using gumbel-softmax, where T is the temperature term (Jang et al., 2017).
PID Policies.
Suppose we know a priori that PID control is suitable for stabilising of an RL system.
We can express this knowledge using the DSL for PID functions that allows C to be expanded as discretized, multivariable PID controllers (Zheng et al., 2002).
We leave the details in Appendix K.2.
Program Interpretability.
Our RL algorithm searches over a DSL to synthesize programmatic policies.
Thanks to their structured and symbolic representation, the algorithm learns highly interpretable policies.
For example, consider an Ant Cross Maze environment depicted in Fig.
The maze contains three possible goal positions and one would be randomly selected at each time.
In this environment, the task for a quadruped MuJoCo
Ant is to reach the selected location by navigating through the maze staring from an initial position on the bottom and without collision or crash.
We consider the DSL for this task using ensemble policies C π .
It includes four basic primitive functions for moving the Ant up π UP , down π DOWN , left π LEFT , and right π RIGHT (pretrained as neural network policies using standard RL algorithms with details left in App. F.2).
Fig.
3 depicts a synthesized program P cross with three branches for solving the Ant Cross Maze environment.
As speciﬁed in Equation 1, our semantics of a branching construct is approximated by the sigmoid function σ .
The value of the predicate in a Boolean condition determines the activation of the controller guarded by the Boolean condition.
At each state, branch activation determines the strength of each of the controllers in the program.
For example, the activation of branch 1 is σ ( θ 1 c
+ θ T 1 ·X ) , and the activation of branch 2 is (1 − σ (
θ 1 c
+ θ T 1 ·X ))
· σ ( θ 2 c + θ T 2 ·X ) .
[Figure]
Fig.
4a depicts the activation of branch 1 as a function of ( x, y ) when the goal to reach is sampled at
G x = 12 , G y = 0 .
The degree of activation (yellow) is close to 1 on all states under (12, 0) indicating that the ensemble policy at branch 1 is used to drive the Ant up to the goal.
Indeed, according to the distribution of each primitive function at branch 1, the effect of π UP dominates.
Fig.
4b, Fig.
4c, and
Fig.
4d depict the activation of all three branches when the goal is at G x = 6 , G y = − 6 .
The program can be interpreted as branch 1 (where π UP dominates) and branch 3 (where π RIGHT dominates) are activated in the yellow areas of Fig. 4b and Fig.
4d respectively.
This allows the Ant to make a curved up and right move to the goal (branch 2 is not activated during execution for this goal).
Problem Formulation.
We frame programmatic RL as a Markov Decision Process (MDP) deﬁned by a tuple {S , A , T , R} where S and A represent the environment state space and action space,
T : S ×
A × S → [0 , 1] captures the set of transition probabilities, and R : S ×
A → R denotes the reward function.
We assume S ⊇ R |X∪V| where X is the set of input variables of a composite program (deﬁned by a DSL) and V is the set of input variables of primitive functions.
For an afﬁne policy, V = ∅ .
At time t ≥ 0 , an RL agent receives an environment state s t ∈ S and performs an action a t ∈ A selected by its policy π ( a t
| s t ) : S → A .
Based on s t and a t , the agent transits to receive the next state according to the transition model T ( s t +1 | s t , a t ) , and receives the reward
R ( s t , a t ) .
We aim to learn a programmatic policy π in the DSL in Fig.
1 by jointly synthesizing the program’s architecture E and optimizing the program’s parameters θ to maximize the cumulative discounted reward E s 0 ,a 0 ,s 1 ···∼ π (cid:2) (cid:80) ∞
γ t · R ( s t , a t ) (cid:3) where γ ∈ (0 , 1] .
A RCHITECTURE S EARCH FOR P ROGRAMMATIC P OLICIES
Inspired by differentiable neural architecture search e.g. DARTs (Liu et al., 2019b), we relax the policy architecture search space to be continuous.
This amounts to collectively optimizing the probability distribution of all program architectures in the search space and assigning the the highest probability to the architecture that maximizes cumulative MDP reward.
Our algorithm is not speciﬁc to a DSL.
It takes as input any policy DSL with differentiable semantics and conducts policy architecture search on a program derivation tree of the DSL.
Formally, a program derivation tree is T
=
{ V, E} where a node u ∈ V contains partial architectures with missing expressions or a complete architecture permissible by the DSL.
An edge ( u, u E ) ∈
E exists if one can obtain the architectures in u E by expanding a nonterminal E within a partial architecture in u following some DSL production rules.
If more than one rule can be applied to expand the nonterminal
E , u E contains more than one architecture.
Take the P cross program in Fig.
3 as a concrete example:
Fig.
5 depicts a program derivation tree for the DSL in Fig.
1 where a controller C is an ensemble policy.
On the root node 0, we have two choices to expand the initial nonterminal E 1 to either an ensemble policy C 1 or a partial architecture if B 1 then C 2 else E 2 .
Node 1 thus contains two partial architectures.
Formally, we use F ( u E ) to represent the set of architectures on a node u E .
To expand a nonterminal or a missing expression of a partial program architecture, we relax the categorical choice of DSL production rules into a softmax over all possible production rules for the missing expression with trainable weights.
For example, on node 1, the choices to expand E 1 between the ensemble policy C 1 and the conditional branching expression are weighted by the weight matrix w 1 (obtained after softmax) drawn in Fig.
Based on w 1 , we choose to expand E 1 to the conditional branching expression on node 1.
Assume we further expand E 2 on node 1 to a conditional branching expression as well on node 5.
Then again we have two choices to expand the nonterminal E 3 on node 5 weighted by w 2 .
This time we choose to expand E 3 to an ensemble policy C 5 .
Formally, the
A program derivation tree T essentially expresses all possible program derivations up to a certain bound on the depth of program abstract syntax trees.
To train architecture weights, we encode a program derivation tree itself as a differentiable program π T θ,w that takes a state s as input.
Its action output is weighted by the outputs of all programs included in π T θ,w , where w represents program architecture weights and θ includes unknown program parameters of all the mixed programs in the tree.
The semantics computation of an expression (cid:74) E (cid:75) ( s ) in a program derivation tree π T θ,w is delegated to its tree node u E where the nonterminal E is expanded and the categorical choice of expanding E on u E is relaxed to a softmax over all possible choices:
[Equation]
Complexity.
Assume that the root of T hosts the initial DSL nonterminal E T , d is the depth of T , k is the number of DSL production rules, and m is the maximum number of nonterminals in the body of any rules.
The semantics of π T θ,w is deﬁned as (cid:74) π T θ,w (cid:75) ( s ) =
(cid:74) E T (cid:75) ( s ) .
The number of DSL operations (e.g. evaluations of ensemble policies and Boolean conditions) invoked by (cid:74) E T (cid:75) ( · ) is bounded by O (( km ) d ) .
In practice, we optimize the run-time cost of (cid:74) E T (cid:75) ( · ) as discussed in Appendix. F.1.
The parameters w and θ of a program derivation tree π T θ,w can be jointly optimized using any policy gradient methods.
To obtain stochastic policy gradients, π T θ,w ( ·| s ) is encoded as a Gaussian policy where the tree program outputs the action distribution mean.
A separate set of parameters specify the (diagonal) distribution covariance.
In this paper, we consider trust region methods e.g. (Schulman et al., 2015) and aim to maximize the “surrogate” objective function, subject to a constraint on the size of the policy update by δ , where ρ π T θ old ,w old is the discounted state visitation frequency of π T θ old ,w old ,
A π T θ old ,w old is an estimator of the advantage function over a ﬁnite batch of samples from π T θ old ,w old and θ old , w old are policy parameters and architecture weights before the update:
[Equation]
Policy Parameter Optimization.
Our training algorithm is an iterative bilevel optimization procedure.
At training iteration k , we perform two steps.
At the ﬁrst step, we optimize the lower-level program parameters θ with respect to (2), freezing the upper-level architecture weights w :
[Equation]
k
[Equation]
Policy Architecture Optimization.
At the second step, we optimize the upper-level architecture weights w with respect to (2), freezing the lower-level program parameters θ :
[Equation]
[Figure]
Training steps (3) and (4) are alternated across training iterations until reward convergence.
They can be approximately solved using the efﬁcient conjugate gradient algorithm, after making a linear approximation to the objective and a quadratic approximation to the constraint (Schulman et al., 2015).
Upon convergence, based on architecture weights, we obtain a discrete program architecture from π T θ,w replacing each tree node containing multiple architectures with the most likely architecture in a top-down manner.
Finally, we train the parameters in the chosen architecture using RL (Schulman et al., 2015) until convergence from the parameter values learned by the architecture search process.
The algorithmic pseudocode is depicted in Algorithm 1 in the appendix.
P ROGRAMMATIC H IGH - LEVEL P LANNING
We further explore learning programmatic policies for high-level planning tasks with long-horizon and weak reward signals.
For example, consider the Ant navigation task in a ⊃ -maze from (Nachum et al., 2018) in Fig.
6a.
Due to the complex maze shape, distance-based rewards do not lead to solving the problem.
Hierarchical RL (HRL) is an efﬁcient approach to solve long-horizon high-level planning.
HRL uses a high-level policy to generate a sequence of high-level goals and low-level policies to generate sequences of actions to achieve each successive goal.
Existing HRL methods are either model-free e.g. (Nachum et al., 2018; 2019) or model-based e.g. (Roderick et al., 2018;
Jothimurugan et al., 2021).
In a model-free approach, the learning algorithm in Sec.
3 can be applied to train HRL policies as programs.
In this section, we focus on model-based planning that takes advantage of the structure in the model of a high-level planning task to improve learning efﬁciency.
For model-based planning, we abstract over both states and actions to ensure a ﬁnite high-level model.
Recall that functions in the DSL of ensemble policies are assumed to be simple, low-level primitives designed to grasp basic, task-agnostic skills (e.g. Ant moving up and down).
We use these primitives as abstract actions because they abstract low-level agent actions to an abstract space of skills.
Regarding state abstractions, similar to previous works (Abel et al., 2020; Winder et al., 2020;
Jothimurugan et al., 2021), we assume that state abstractions can be effectively provided by domain experts.
For instance, the abstract states of the ⊃ -maze are collected on an abstract 2-D gridworld m ∈ R N × N in Fig.
6b.
An abstract grid ( x, y ) ∈ N × N subsumes concrete positions with a known scale where m [ x, y ]
= 1 indicates walls and m [ x, y ]
= 0 indicates navigable spaces.
The domain expert can then specify a pair of abstract initial state ϕ
Init (yellow) and goal state ϕ Goal (green).
[Equation]
Our design choice of state and action abstraction addresses an important challenge of model-based
HRL — it has to simultaneously perform model-based high-level planning and estimate the state transition probabilities of the abstract model based on the current high-level policy (Jothimurugan et al., 2021).
In our approach, we decouple this dependency.
Since abstract actions as primitive functions implement simple, task-agnostic skills, it is straightforward to directly specify the statetransition behavior of the abstract actions ϕ Act as opposed to learning it.
For example, the Ant primitives move an Ant by updating its horizontal position x i or vertical position y
i one step dx i or dy i on the abstract model.
The Ant can also be moved diagonally via primitive composition:
[Equation]
An abstract action is speciﬁed together with a guard ϕ Guard that encodes the condition under which the action can occur.
For example, an Ant in ⊃ -maze should not walk into walls and the path from an old position to a new position should not be blocked by walls.
[Equation]
We further leverage the primitive function speciﬁcations to synthesize a high-level control plan on the abstract model.
Suppose for now that we are searching over a high-level plan of a ﬁxed number k of abstract action steps to reach the goal state ϕ Goal from the initial state ϕ
Init :
[Equation] [Equation]
Constraint (5) can be solved by off-the-shelf constraint solvers e.g. Z3 (de Moura & Bjørner, 2008).
We incrementally search for longer and longer high-level plans, starting from k = 1 and increasing k until the optimal solution is found.
Such a plan is a sequence of sub tasks.
For example, the solved high-level plan for Ant ⊃ -Maze navigation is depicted in Fig.
6c.
Since a sub task may depend on multiple primitives e.g. Ant turning around at a corner, we learn a low-level policy for each sub task as a programmatic ensemble policy composing primitive functions using the technique in Sec.guided by distance-based rewards.
This ensures that our HRL policies are interpretable at both high level and low level.
More details including the algorithmic pseudocode are given in Appendix.
A.2.
Compared to existing model-based HRL approaches e.g. (Jothimurugan et al., 2021), our algorithm additionally needs abstract actions to be speciﬁed (as ϕ Act and ϕ Guard ) rather than learned.
We show that (1) providing speciﬁcations of task-agnostic abstract actions (primitive functions) is trivial even for complex high-level planning tasks (Appendix G.1); (2) more importantly, these easy-to-annotate speciﬁcations substantially improve the sample efﬁciency of HRL (Sec. 5).
E XPERIMENTS AND E VALUATIONS
We evaluated our approach on two groups of challenging continuous control benchmarks involving motion control and task planning 1 .
Group one contains four MuJoCo environments that require agents to reach or move an object to target locations: (1) Ant Cross Maze : the example depicted in
Fig.
(2) Ant Random Goal : The quadruped MuJoCo Ant in Fig.
10a is trained to reach a randomly sampled goal location within a conﬁned circular region.
(3) Pusher : A robotic arm in Fig.
10b is trained to push a cylinder object to a given target location.
(4) HalfCheetah Hurdle :
A MuJoCo halfcheetah in Fig. 10c is required to run and jump over three hurdles to reach a given goal area.
Group two consists of three hierarchical RL benchmarks: (1) Ant ⊃ -Maze : the example depicted in
Fig.
6a.
(2) Ant Push : This task requires the Ant in Fig.
10d to push away a movable block to reach the goal region behind it.
(3) Ant Fall : The Ant in Fig.
10e is required to push a movable block into a rift to ﬁll the gap and then walk across it to reach the target on the other side of the rift.
Multiple neural network primitives were trained and included in our ensemble policy DSL
e.g. HalfCheetah π JUMP and π FORWARD , Pusher π PUSH-LEFT and π PUSH-DOWN (with details in Appendix F.2).
Programmatic RL.
To the benchmarks from the ﬁrst group, we apply the RL algorithm in Sec.
3 to learn a program derivation tree π T -PRL from which we extract programmatic RL policies π -PRL.
The tree depth bound of π T -PRL is set to 6.
Our baselines include Composition-SAC (Qureshi et al., 2020) that learns Bidirectional LSTM-based ensemble policies using the SAC algorithm (Haarnoja et al., 2018) to combine the task-agnostic primitives for solving the task environments.
On-policy
RL algorithms TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) are the other two baselines.
Fig.
7a depicts the mean learning performance in terms of the agent’s ﬁnal distance from targets over 10 random seeds.
TRPO and PPO both fail to reach the goals.
During policy architecture search, π T -PRL successfully solves all tasks and outperform or match the performance of the BiLSTM models of Composition-SAC.
The π T -PRL models exhibit less data-efﬁciency than the Composition-SAC policies in part due to the use of TRPO for on-policy update that is less sample-efﬁcient than off-policy SAC (our learning algorithm can also be applied to off-policy RL).
Upon convergence of training a π T -PRL program derivation tree, our algorithm extracts a program π -PRL from π T -PRL and trains it until convergence.
Table 1 presents the mean and standard deviation of π -PRL’s ﬁnal distance from given targets.
Extracted programs recover the performance of π T -PRL models and continue to outperform or match the BiLSTM models in Composition-SAC.
Importantly, π -PRL policies are signiﬁcantly more interpretable than the BiLSTM policies e.g. Fig.and Fig.
The high interpretability of π -PRL policies in turn leads to strong generalizability to novel environments because of the inductive bias implicitly encoded in the policy’s interpretable
[Table]
representation, e.g. if-then-else conditions used to drive the Ant to different regions on a maze.
We show more examples about the interpretability and generalizability of π -PRL in Appendix I and J.
In Table 1, we also show the number environment steps needed until convergence for policy architecture search π T -PRL, extraction π -PRL
[Table]
and training a programmatic policy using the same architecture as π -PRL from scratch .
Except Ant Cross Maze, the sample efﬁciency of policy architecture search and extraction combined is better than learning a single program from scratch.
Intuitively, this is because π T -PRL is essentially an over-parameterized model of the optimal policy in the search space.
The better data efﬁciency of our method compared to enumerating all possible programs to perform RL for each highlights
the merits of jointly optimizing policy architectures and parameters.
Programmatic High-level Planning .
To the second group of the benchmarks, we apply the highlevel planning algorithm in Sec.
4 to learn hierarchically programmatic RL policies, denoted as π -HPRL programs.
Our baselines include a model-free Hierarchical RL method HIRO (Nachum et al., 2018) and a model-based high-level planning method A-AVI (Jothimurugan et al., 2021).
All algorithms run under the same setting to reach a given target, which is ﬁxed for each environment.
Fig.
7b depicts the mean learning performance in terms of the agent’s ﬁnal distance from targets over 5 random seeds.
The π -HPRL programs are more stable, outperform the baselines, and uniquely solve the Ant Push and Ant Fall environments.
We show the success rates of each algorithm in
Appendix G.2.
Fig.
7b shows that HIRO gets stuck in local optimums.
Unlike π -HPRL, HIRO is model-free and does not know the structure of the abstract state space, so it is unable to discover the path from the initial region to the goal region.
A-AVI performs worse than π -HPRL because it does not decouple model-based high-level planning and model construction.
π -HPRL is more sample efﬁcient as it decomposes a task into multiple simpler sub tasks with explicit sub goals to
[Table]
reach in an optimal high-level plan.
First, since the high-level plan is in accordance with the primitive speciﬁcations, a sub goal is easy to achieve by utilizing the corresponding primitives.
Second, the training procedure can be guided by dense distance-based rewards to the sub goal.
Therefore, training for each sub task converges extremely fast well in advance of the expiration of the training budget allocated to the sub task, causing the ladder-shape convergence curves in Fig.
7b
[Table]
Ablation Study .
We investigate the impact of compositionality in our algorithm.
For the ﬁrst group of our benchmarks, we apply our algorithm to synthesize programs in the DSL in Fig.
1 that switch back and forth between a set of afﬁne controllers under different conditions.
Table 2 shows that these programs perform much worse than the composite programs, which highlights the merits of composition.
We additionally study the effect of the depth bound on a program derivation tree for policy architecture search.
We set the depth bound to 5, 6, 7 respectively.
Table 2 shows that the ﬁnal performance of the synthesized policies is not sensitive to these settings.
Our algorithm converges to similar architectures as it strives to assign the highest probability to the architecture that maximizes cumulative RL reward.
Convergence curves of all the ablated versions are depicted in Appendix B.
Table 2 also reports the performance comparison between our oracle-free programmatic RL algorithm with an oracle-guided baseline (Verma et al., 2018).
The baseline learns programmatic policies that imitate the neural Compositional-SAC policies in Table 1.
To ensure a fair comparison, we relax the semantics of these programs to be continuous as well.
The results show that the performance of the distilled programs is worse than that of their oracles and our policies.
Since the neural oracles and programs reside in very different policy architecture spaces, the program that best imitates an oracle is not necessarily a performant programmatic policy.
Comparison results with other oracle-based programmatic RL baselines can be found in Appendix C.
R ELATED W ORK AND C ONCLUSION
Existing programmatic RL methods mostly train a programmatic policy to imitate a pretrained
RL oracle.
They synthesize policies by enumerating a set of templates in the form of decision trees (Bastani et al., 2018; Silver et al., 2020), ﬁnite state machines (Inala et al., 2020), or program sketches (Verma et al., 2018; 2019).
However, imitation-based approaches suffer from a nontrivial distillation gap as the distillation process can yield suboptimal policies.
Our approach optimizes programmatic polices in a continuous relaxation of the non-differentiable architecture space solely using reward signals by policy-gradient methods.
Trivedi et al.
ﬁrst learns a smooth program embedding space and then searches over the embedding space to synthesize a program.
Our approach differs as we do not need to prepare a dataset of programs to train the program embedding space, thus is more suitable when sampling such a dataset is challenging.
Yang et al.
(2021) uses a MaxSAT solver to synthesize straight-line programs to guide a policy to reach a goal.
Our method is more suitable when there are multiple goals for which architecture synthesis is necessary to learn conditional branches to reach different goals.
We discuss other related work in broader contexts in Appendix.
E.
Conclusion.
We present a novel programmatically interpretable and compositional RL framework.
Our method jointly learns policy parameters and policy architectures in a continuous relaxation of the non-differentiable program architecture space via policy-gradient methods.
Our RL framework leverages compositionality in programming languages to integrate primitive functions into a composite program to solve novel RL problems.
Experiment results demonstrate that it excels in discovering compositional RL programs with optimal architectures and strong interpretability.
R EPRODUCIBILITY S TATEMENT
We have included instructions to reproduce our results in the supplementary material.
The code of this work is available at https://github.com/RU-Automated-Reasoning-Group/ pi-PRL .
A CKNOWLEDGMENTS
We thank the anonymous reviewers for their comments and suggestions.
This work was supported by
NSF Award #CCF-2124155 and NSF Award #CCF-
A T RAINING A LGORITHM P SEUDOCODE A. T HE P ROGRAMMATIC R EINFORCEMENT L EARNING A LGORITHM
We depict the pseudocode of our oracle-free programmatic reinforcement learning algorithm in
Algorithm 1.
The algorithm builds a program derivation tree that involves programs whose abstract syntax tree depth is bounded by a hyperparameter d .
We encode a program derivation tree as a differentiable program π T θ,w with architecture weight w and policy parameters θ .
Recall the surrogate objective deﬁnition in Eq.
[Equation]
where A π T θ old ,w old is an estimator of the advantage function over a ﬁnite batch of samples from π T θ old ,w old and θ old , w old are policy architecture weights and parameters before policy update.
In the pseudocode, training steps (3) for learning θ and (4) for learning w are alternated across training iterations until reward convergence.
They can be approximately solved using the efﬁcient conjugate gradient algorithm, after making a linear approximation to the objective and a quadratic approximation to the constraint (Schulman et al., 2015).
Upon convergence, based on architecture weights, we obtain a discrete program architecture from π T θ,w replacing each tree node containing multiple architectures with the most likely architecture in a top-down manner.
Finally, we train the parameters in the chosen architecture using RL (Schulman et al., 2015) until convergence from the parameter values learned by the search process.
Algorithm
Require: Policy Context-free grammar E , depth bound d , KL-divergence limit δ
Ensure: Synthesized programmatic policy π
T ← get Program Derivation Tree ( E, d ) (cid:46) T expresses all programs whose abstract syntax tree depth is bounded by d π T θ,w ← encode ( T ) (cid:46) π T θ,w as differentiable program with parameters θ and architecture weights w that encodes T θ 0 , w 0 ← randomly initialize ( θ , w ) for k = 0 , 1 , 2 , . . .
, M do (cid:46) Policy architecture search θ k +1 = arg max θ J θ k ,w k
( θ, w k ) s.t.
E s ∼ ρ π T θk,wk (cid:104) D
[Equation]
( π T
[Equation]
k ,
[Equation]
k ( ·| s ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
( ·| s )) (cid:105) ≤ δ (cid:46) Per Eq.
(3) w
[Equation]
arg
[Equation] [Equation]
, w ) s.t.
E s ∼ ρ
[Equation]
θk
[Equation]
,wk (cid:104) D
[Equation]
( π
[Equation]
k
[Equation] [Equation]
( ·| s ) (cid:12)(cid:12) (cid:12) (cid:12)(cid:12) (cid:12) π T θ k +1 ,w k +1 ( ·| s ))
(cid:105) ≤ δ (cid:46) Per Eq.
(4) end for π θ ← extract ( π T θ M ,w M ) (cid:46) Extract a single program based on architecture weights w M θ 0 ← θ M (cid:46)
The extracted policy inherits parameters from trained π T θ M ,w M for k = 0 , 1 , 2 , . .
. , N do (cid:46)
Train the chosen architecture θ k +1 = arg max θ J θ k ( θ ) s.t.
E s ∼ ρ πθk (cid:104) D KL ( π θ k
( ·| s ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
π θ k +1 ( ·| s ))
(cid:105) ≤ δ end for return π θ N
A. T HE P ROGRAMMATIC H IGH - LEVEL P LANNING
The pseudocode of the synthesis algorithm for HRL policies is depicted in Algorithm 2.
The goal is to synthesize a high-level control plan on the abstract model using abstract actions ϕ Act as constraints.
An abstract action is speciﬁed together with a guard ϕ Guard that encodes the condition under which the action can occur.
Suppose for now that we are searching over a high-level plan of a ﬁxed number
[Table]
[Equation]
[Table]
∧
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Constraint (6) can be solved by off-the-shelf constraint satisﬁability solvers e.g.
Z3 (de Moura & Bjørner, 2008).
We incrementally search for longer and longer high-level plans, starting from k = 1 and increasing k until the optimal highlevel plan solution is found.
A solved high-level plan is a sequence of sub-tasks: (cid:0) β ( x 0 , y 0 ) , π 1 , β ( x 1 , y 1 ) (cid:1) , (cid:0) β ( x 1 , y 1 ) , π 2 , β ( x 2 , y 2 ) (cid:1) , . . .
, (cid:0) β ( x k − 1 , y k − 1 ) , π k , β ( x k , y k ) (cid:1) where we extract the intermediate high-level goals x 1 , y 1 , . . .
, x k − 1 , y k − 1 from the solution of the statisﬁability solver, and the concretization function β converts abstract states back to concrete goals on the unabstracted environment for the agent to successively achieve.
For example, the solved high-level plan for Ant ⊃ -Maze navigation is depicted in Fig.
6c.
For each sub-task i that starts from a set of initial states β ( x i − 1 , y i − 1 ) , we learn a new low-level policy π i that drives the agent to reach states in β ( x i , y i ) .
The states of π i when reaching β ( x i , y i ) serve as the initial states for training π
i +1 .
Notice that we do not directly use primitive functions as π i , because oftentimes low-level policies need to compose primitive functions using different weights under different environment conditions to achieve a sub-task, e.g. Ant turning around at a corner with different poses and velocities while contacting walls in the process.
We therefore learn each low-level policy
π i as a programmatic ensemble policy composing primitive functions using the technique in Sec. 3 and distance-based rewards.
This ensures that our HRL policies are interpretable at both high level and low level.
Algorithm
Require: Primitive Speciﬁcations ϕ Act , abstract Action Guard ϕ Guard , abstract initial ϕ init and goal ϕ goal , time-step limit T for the execution of each low-level policy
Ensure: A high-level plan model , and its low-level policies π 0 , π 1 , . . .
, π k −for k = 1 , 2 , . . .
, M do (cid:46) High-level policy planning φ k ← ϕ Init ( x 0 , y 0 )
∧ (cid:86) ki =1 (cid:0) ϕ Act ( x i − 1 , y i − 1 , x i , y i )
∧ ϕ Guard ( x i − 1 , y
i − 1 , x i , y i ) (cid:1)
∧ ϕ Goal ( x k , y k ) (cid:46) Per Eq.
(5) model ← solve ( φ k ) if model (cid:54) =
None then ∀ i. x i , y i = model
[ x i ] , model [ y i ] (cid:46)
A high-level control plan is synthesized
Break else if k ==
M then
Return “Failed” end if end for for i = 0 , 1 , 2 , . . .
, k − 1 do (cid:46) Low-level policy learning
R i ← λs, a. − dist ( s, β (cid:0) x i , y i ) (cid:1) (cid:46) Reward based on distance to sub goal ( x i , y i )
π
i = arg max π i E s 0 ∼ (cid:0) π 0 ,...,π i − 1 (cid:1)
[Table]
return model , ( π 0 , π 1 , . .
, π k − 1 )
B M ORE A BLATION S TUDY R ESULTS
Our ﬁrst ablation study investigates the impact of compositionality on our algorithm.
Fig.
8 shows the ablation study results on comparing different strategies, i.e., learning a programmatic ensemble policy, learning a programmatic afﬁne policy, and learning a deep neural network policy using TRPO (Schulman et al., 2015).
A programmatic afﬁne policy is a monolithic program in our DSL (Fig. 1) that switches back and forth between a set of afﬁne controllers under different environment conditions.
The result shows that our programmatic ensemble policies signiﬁcantly outperform the other two representations in all group one environments, which highlights the merits of composition.
Notice that in the HalfCheetah Hurdle and Pusher environments, the programmatic afﬁne policies signiﬁcantly outperform the neural network policies.
Our second ablation study investigates the impact of the depth bound of a program derivation tree on the performance of our algorithm.
Fig.
9 shows the ablation study result about using different depth bounds on program derivation trees and compares the mean normalized ﬁnal distances to goals of learned policies with standard deviations (averaged over 10 random seeds).
Speciﬁcally, we compare program derivation trees with depths of 5, 6 and 7.
The result demonstrates that the training performance is not sensitive to these depth bounds.
The algorithm converges to architectures with similar performance.
Table 2 additionally shows that the structures of the converged architectures are also similar.
This suggests that if the user does not know how complex a programmatic policy should be a priori, the user can simply set a large tree-depth bound and let the learning algorithm assign the highest probability to the architecture that leads to a program maximizing cumulative RL reward.
C C OMPARISON WITH O RACLE - GUIDED P ROGRAMMATIC RL
One of our main contributions is direct programmatic policy search.
Existing oracle-guided programmatic RL algorithms suffer from a nontrivial distillation gap as the distillation process can yield suboptimal programmatic policies whose reward performance is signiﬁcantly worse than that of their oracles.
This is because oracles (e.g. neural policies) and programs may reside in very different policy structure spaces.
Due to the structural constraints, the program that best imitates an oracle is not necessarily a performant programmatic policy, or could even be much worse than the optimal program in the search space.
To address this limitation, we propose programmatic RL solely based on reward signals without using any oracles.
This allows the synthesis algorithm to search for optimal programs in the programmatic policy space.
We compare our programmatic RL algorithm with oracle-based programmatic RL algorithms that learn policies in the form of programs
[Table]
(using the same DSL as ours) (Verma et al., 2018) and decision trees (Bastani et al., 2018).
In Table 3, we report results on Mujoco/OpenAI environments where programmatic policies invoke a set of afﬁne controllers under different environmental conditions.
To ensure a fair comparison, we relax the semantics of oracle-guided RL programs to be continuous as well but decision trees remain discrete.
Neural oracles for Walker2d, Hopper, HalfCheetah, and Ant were trained with SAC (Haarnoja et al., 2018) using 3 million environment steps.
Neural oracles for Reacher, Swimmer, BipedalWalker, and
Pendulum were trained with TRPO (Schulman et al., 2015) using 3 million environment steps and we report the averaged ﬁnal reward performance of three repeated experiments.
The reward performance of our oracle-free programmatic RL policies is comparable to neural network policies.
Due to the distillation gap, the oracle-guided RL programs learned by the baselines perform worse than their oracles and our policies.
We also tried the imitation-projected programmatic RL algorithm (Verma et al., 2019) that optimizes a programmatic policy by taking a mirror gradient descent in a policy space with a mix of neural and programmatic representations, which enables deep policy gradient approaches on programmatic policies.
However, our own implementation of the algorithm in the mixed policy space does not lead to signiﬁcant policy improvement compared with (Verma et al., 2018).
We further compare our algorithm with Verma et al.
(2018) on the environments reported in Sec. 5.
According to our results in Fig.
7a, these environments are best solved using policies that invoke a set of ensemble controllers.
We report the mean normalized ﬁnal distances to goals of learned policies with standard deviations.
The neural oracles were trained by Compositional-SAC (Qureshi et al., 2020).
Table 2 demonstrates that the oracle-guided RL programs ( π -Oracle) are again suboptimal due to the distillation gap.
Especially, they do not work well in environments with multiple goals.
For example, we found that although on Ant Cross Maze the neural compositional-SAC policy in general works well, the success rate of reaching the goal (6, -6) is higher than reaching the other goals.
The bias is ampliﬁed in the oracle-guided programmatic RL policy.
When the goal is at (12, 0), the agent sometimes mistakenly goes somewhere near (6, -6).
The above two sets of experiments demonstrate that our oracle-free programmatic RL overcomes the suboptimality induced by oracle-guided programmatic RL.
D DSL S WITHOUT L OOPS
Our loop-free programmatic policies can already produce repetitive behaviors when necessary.
This is because policies are always executed in a feedback loop with RL environments.
At each iteration, environment states encountered by repetitive behaviors would repeatedly activate or be matched to the same if-then-else conditions and therefore trigger alike actions.
To support this argument, we evaluated our tool on two environments from Inala et al.
(2020): Car and QuadPO.
Both environments need policies that capture repeating behaviors.
For Car, the goal is to drive a car out of a parking spot to an adjacent lane while avoiding collisions.
For QuadPO, the goal is to maneuver a 2D quadcopter through an obstacle course by controlling its vertical acceleration.
The training and test distributions are varied to evaluate whether policies can produce an arbitrary number of repetitions.
For example, [Table] on QuadPO, obstacle course
Table 4 summarizes by measuring the fraction of rollouts (out of 500) that safely reach the goal
Our loop-free programmatic policies generalize better than neural policies on the test distributions. We did not compare our policies with the state-machine policies (Inala et al., 2020) as its implementation is not available.
Moreover, our tool indeed supports loops in a program that sequentially processes a history of environment states and actions.
For example, our DSL allows a controller C to be expanded as a discretized, multivariable PID controller in Fig.
In Appendix K.2, such a PID controller is formalized based on the higher-order combinator fold that acts over a ﬁxed-sized window on a history of observations.
We could alternatively add fold ( f, h ) directly to the DSL to search policies that combine the results of recursively processing each past observation in a history h to build up a return action (the body of the combining operation f can also be synthesized).
E A DDITIONAL R ELATED W ORK
Compositional Reinforcement Learning.
To improve RL efﬁciency, a line of research that transfers past skills into new skills for solving new complex problems has emerged.
Lee et al.
(2019; 2020) learn high-level polices to select the previously trained primitive policies.
The macro-action models (Vezhnevets et al., 2016) similarly learn a high-level planner to combine actions sequences from primitives.
These methods activate only one policy at each timestep.
Other works have explored simultaneously composing multiple task-agnostic primitive skills by learning to assign weights for action composition (Peng et al., 2019; Qureshi et al., 2020).
Hierarchical reinforcement learning, such as option-critic (Bacon et al., 2017), is also a promising approach to solve complex tasks.
However, option-critic is prone to inefﬁcient task decomposition.
Nachum et al.
(2018) address this issue by automatically decomposing a complex task into subtasks and solving them by optimizing the subtask objectives.
As opposed to these efforts, our method learns high-level and low-level policies that are both interpretable.
Our technique can integrate any primitive skills diverse enough to complete a task and is orthogonal to automatic skill discovery methods e.g. (Eysenbach et al., 2019).
Differentiable Architecture Search.
Neural architecture search has emerged as a promising approach to automate deep learning applications (Zoph & Le, 2017; Liu et al., 2018; Real et al., 2019).
Particularly, our program architecture synthesis algorithm is inspired by DARTS (Liu et al., 2019b).
This method uses a composition of softmaxes over all possible candidate operations between a ﬁxed set of neural network nodes to relax the neural architecture search space in a super-network, and uses approximate gradient descent to iteratively train the weights and parameters of the super-network.
It then selects the optimal architecture based on the weights.
Various methods further improve architecture search efﬁciency, accuracy and applicability (Chen et al., 2019a;
Guo et al., 2020; Zhao et al., 2021;
Wang et al., 2021; Cui & Zhu, 2021; Jiang et al., 2019; Liu et al., 2019a).
However, none of these methods has performed differentiable architecture search in a reinforcement learning context or considered leveraging symbolic logical abstractions for hierarchical architecture synthesis.
Neural Program Synthesis.
Neural program synthesis involves learning neural networks to predict function distributions to guide a synthesizer e.g. DeepCoder (Balog et al., 2017), or generate a program autoregressively in an end-to-end fashion e.g. (Sun et al., 2018; Parisotto et al., 2017; Bunel et al., 2018) and RobustFill (Devlin et al., 2017).
BUSTLE (Odena et al., 2021) uses bottom-up search that allows the model to prioritize and combine small programs that solve different subtasks.
SKETCHADAPT (Nye et al., 2019)
ﬁrst generates a program sketch with holes, and then ﬁlls the holes using a conventional synthesizer.
BAYOU (Murali et al., 2018) infers a different form of program sketches that abstract names and operations by their type.
DreamCoder (Ellis et al., 2021)
[Table]
iteratively builds sketches using progressively more complicated primitives though a wake-sleep algorithm. Latent Programmer (Hong et al., 2021) considers two-level search for program synthesis, in which the synthesizer ﬁrst generates a plan — a sequence of symbols, in other words discrete latent code from input/output examples by discrete autoencoders, that describes the desired program at a high level, and then generates the program in the target language.
These methods avoid enumerating every possible program, which is prohibitively expensive for large program spaces.
There is also a line of work that deals with learning to process partial programs in addition to the speciﬁcation.
In execution-guided program synthesis, the model guides iterative expansions of a partial program until a matching one is found.
Zohar & Wolf (2018) process intermediate values of a program using a neural network for search direction prioritization for a small, straight-line
DSL.
Ellis et al.
(2019) allow values of whole and ground programs encountered during bottom-up search to be used to prioritize the search in an actor-critic framework.
Similarly, Chen et al.
(2019c) exploit intermediate values while synthesizing a program via a top-down fashion using a neural encoder-decoder model.
Tian et al.
(2019) exploit the differentiability of a rendering process to train a program generating policy and generalize it beyond the training distribution via gradient-based ﬁne-tuning.
F I MPLEMENTATION D ETAILS
In this section, we present implementation details including the optimization of (cid:74) E (cid:75) ( · ) , primitive policies, symbolic inputs and reward structures of each environment, and hyperparameters of reinforcement learning algorithms.
F. C OMPUTATIONAL C OMPLEXITY OF P ROGRAM D ERIVATION T REE E XECUTION
We compare the computational complexity of (cid:74) E (cid:75) ( · ) (where d = 6 ) with that of the deepest single program in (cid:74) E (cid:75) ( · ) and that of the Bidirectional LSTM-based model used by the Compositional SAC baseline (Qureshi et al., 2020) as follows.
We report the average running time of over 10000 random executions of the three models in Table 5.
In our implementation, the run-time execution of a program derivation tree is optimized as follows.
As depicted in Fig.
5, primitive policies are invoked multiple times by various programs embedded in a program derivation tree.
Since the input to any primitive is always the current environment state, it sufﬁces to call each primitive just once and use its result anywhere it is invoked in the tree policy.
Table 5 shows that this simple optimization could enable us to run our algorithm as fast as the
Compositional SAC baseline that does not perform any architecture search.
More importantly, we do not need to explicitly enumerate and execute all programs up to some depth for each state evaluated during an episode.
Fig.
5 shows that we allow the exponential number of programs in a program derivation tree to share computation (similar to weight sharing in DARTS) — intermediate results computed by a shallower program can be reused by a deeper program.
For more complex DSLs with a large number k of production rules, we could optimize the architecture search procedure by applying a strategy similar to Progressive DARTS (Chen et al., 2019b).
This strategy gradually increases tree depth d during search while dropping the search directions leading to lowest-weighted programs at the previous stage from the program derivation tree.
Such a progressive procedure would allow our algorithm to deeply explore the architecture space even when k is large, which is left for future work.
F. P RIMITIVE P OLICIES D ETAILS
In all reinforcement learning tasks, we require multiple primitive skills policies to compose higherlevel programs.
Since primitive policies are obtained for completing dedicated tasks, e.g., navigating the ant agent to move up, or pushing down the object in the Pusher environment, it is unnecessary to include targets information in high-level tasks into their observation space during training.
All primitive skills policies are obtained using SAC algorithm (Haarnoja et al., 2018) implemented in
OpenAI Spinning Up RL framework (Achiam, 2018).
Ant.
For standard-torque and low-torque ant, we train two sets of four basic primitive policies, i.e.,
UP , DOWN , LEFT and RIGHT for 1 million steps for each.
The reward function to acquire above four primitives is deﬁned as:
[Equation]
where v direction is the velocity of given direction, I ( IsHealthy ) is a Boolean function that determines the health condition of the agent, action a is a 8-dimension vector, and f contact is a vector that encodes the contact force of the agent.
In order to train UP and DOWN primitives, v direction = ± v x should track velocities on ± x -axis; similarly, to train LEFT and RIGHT policies, we should set v direction = ± v y to keep track of velocities along ±
y -axis.
Coefﬁcients c v , c h , c a and c f are set to be 1, 1, 0.5, 5 × 10 − 4 , respectively.
Pusher.
The Pusher environment requires two skills: PUSH-DOWN and PUSH-LEFT used to manipulate the object to move down and left, and both of them are trained for 0.5 million steps.
The reward function to acquire these two primitives is deﬁned as:
[Equation]
where v direction denotes the object’s velocity projected on DOWN (negative y -axis) or LEFT (positive x -axis), and a is the action.
Coefﬁcients c v and c a are set to be 1 and 0.1, respectively.
HalfCheetah.
In HalfCheetah Hurdle environment, two simple policies JUMP and FORWARD are trained for 0.5 million steps, and they are sufﬁcient to compose the program and solve the task.
The reward function can be deﬁned as:
[Equation]
(9) where v forward and v jump are velocities projected on forward ( x -axis) and jump ( y -axis) directions, and a is the action.
In order to train FORWARD policy, coefﬁcients c v f , c v j are set to be 1 and 0; for policy JUMP , we set coefﬁcients c v f = 0 .
05 and c v j = 1 ,
respectively.
Coefﬁcient c a is the control cost coefﬁcient which set to be 0.1.
Note that for Ant and HalfCheetah Hurdle environment, primitive policies are trained in a blank environment without any obstacle so that they can move freely.
F. P ROGRAM I NPUTS
For low-torque ant, we consider its current position x, y , distance from current position to the starting point (cid:107) x, y (cid:107) 2 , and the angle between current direction to the x-axis arctan yx as symbolic inputs.
For standard-torque ant, besides the above inputs, we also take the goal position G x , G y for navigation, since goals in environments which utilize the standard ant are randomly sampled.
In the HalfCheetah
Hurdle environment, the position of the next hurdle H next and the distance from HalfCheetah’s back foot to next hurdle | x back − H next | are considered.
For the simple pusher task, we use distance from origin to the object (cid:107) x obj , y obj (cid:107) 2 and to the robotic arm (cid:107) x arm , y arm (cid:107) 2 as symbolic inputs. [Figure]
F. E NVIRONMENT D ETAILS F.4. A NT E NVIRONMENTS
The standard-torque Ant environments include Ant Cross Maze and Ant Random Goal environments.
In these environments, a MuJoCo quadruped ant with 150 units torque limit is required to start from origin and reach the random given targets.
The radius of the goals are 1 and 0.25, respectively.
Speciﬁcally, the goal of Ant Cross Maze is randomly sampled from one of three positions: (6, 6), (12, 0) and (6, -6) at each time; and the goals in Ant Random Goal is a random position within a circle with radius of 5.
In ant environments, the reward function is deﬁned as:
[Equation]
(10) where p ant =
( x ant , y ant ) and p goal = ( x goal , y goal ) are positions of ant and the given targets,
I ( IsHealthy ) is a Boolean function that determines the health condition of the agent, action a is a vector, and f contact is a vector denotes the contact force.
Coefﬁcients c distance , c d , c h , c a
and c f are set to be 1, 5, 0.05, 0.01 and 0.001, respectively.
Additionally, we consider three challenging low-torque MuJoCo
Ant environments with weak distance-based reward signals.
Compared to standard-torque, the low-torque ant has the force range [-30, 30].
Since smaller force is exerted on joints, the ant moves slower and suitable for intricate HRL tasks.
Speciﬁcally, we normalize the distances to a ﬁxed ﬁnal goal by the agent initial distance from the goal so values close to 1 or higher show failure.
Greedily following this kind of reward signals does not lead to solving the problems as these environments are in complex shapes.
In the Ant ⊃ -Maze environment, we place the Ant in a ⊃ -shaped maze for a navigation task.
The agent is trained to reach the farthest end of the maze located at (0, 19).
In the Ant Push environment, the moveable block is at (0, 8), and the goal is at (0, 19).
In the Ant Fall environment, the movable block is at (8, 8) at the same elevation as Ant.
Their is a rift in the region [-4, 12] × [12, 20].
To reach the target on the other side of the rift, the Ant must push the block down into the rift, and then step on it to get to the goal position.
For all three environments, the low-torque Ant starts from the origin, and the radius of goals are set to be 1.
F.4. P USHER E NVIRONMENT
In the Pusher environment, the initial position of the object is randomly sampled from the region [0.3, 1] ×
In order to successfully move the object to the given target centered at (-1, 0) with radius of 0.17 using the robotic arm, the distance from arm to object, as well as object to goal should be tracked.
The reward function for Pusher environment is deﬁned as:
[Equation]
a (cid:107)(11) where p arm =
( x arm , y arm ) , p obj =
( x obj , y obj ) , and p goal = ( x goal , y goal ) are positions of the robotic arm, object, and the goal, respectively.
Coefﬁcient c reach , c move , c
a are set to be 0.5, 1, and 0.1, respectively.
F.4. H ALF C HEETAH H URDLE E NVIRONMENT
In the HalfCheetah Hurdle environment, a MuJoCo halfcheetah is required to start from origin and jump over three hurdles to reach the target with radius of 1.
The reward function is deﬁned as:
[Equation]
where x is cheetah’s current position, x goal is target location, count ( x ) function returns the number of hurdles has been passed, I ( IsReached ) is a Boolean function that determines if the cheetah reach the target, v x and v y are the projected velocities on x -axis
and y -axis,
and I ( IsCollided ) is a Boolean function that determines if the cheetah is in collision.
Coefﬁcient c distance , c hurdle , c goal , c run , c jump and c collision are set to be 0.1, 1, 1000, 1, 0.3 and 2, respectively.
F. H YPERPARAMETERS
Following hyperparameters are used to train primitive policies with SAC (Haarnoja et al., 2018) algorithm.
• Discount factor γ = 0.99.
• SGD optimizer; actor learning rate 0.001; critic learning rate 0.001.
• Mini-batch size n = 100.
• Replay buffer of size 100000.
• Soft update targets τ = 0.005.
• Target update interval and gradient step are set to be 1.
Following hyperparameters are used to train π - PRL programs for solving tasks in group one environments with TRPO (Schulman et al., 2015) algorithm.
• Discount factor γ = 0.99.
• Number of trajectories per epoch N = 50 .
• Maximum search depth of program derivation graph D g = 6 .
• KL-Divergence limit δ = 0.01.
• GAE λ = 0.97.
• Gumbel-Softmax Temperature T = 0.25.
Following hyperparameters are used to train π - HPRL programs for solving tasks in group two environments with TRPO algorithm.
• Discount factor γ = 0.995.
• Number of trajectories per epoch N = 100 .
• KL-Divergence limit δ = 0.01.
• GAE λ = 0.97.
• Gumbel-Softmax Temperature T = 0.25.
Table 6 summarizes the neural network architectures used in reinforcement learning algorithms. [Table]
G I MPLEMENTATION D ETAILS OF P ROGRAMMATIC H IGH -L EVEL P LANNING G. S PECIFICATIONS FOR H IGH -L EVEL P LANNING
In our experiment, we have applied programmatic high-level planning in Sec.
4 to 3 challenging environments, namely Ant ⊃ -Maze (Fig. 6a), Ant Push (Fig. 10d), and Ant Fall (Fig. 10e), that are commonly used to evaluate the effectiveness of hierarchical reinforcement learning.
Sec.
4 shows how to perform high-level planning based on the ensemble policy DSL and uses Ant ⊃ -Maze as an example.
In this section, we apply programmatic high-level planning to solve Ant Push and Ant Fall.
Recall that our method builds an abstract high-level model based on state abstractions provided by a domain expert, similar to previous works (Abel et al., 2020; Winder et al., 2020; Jothimurugan et al., 2021).
For example, consider the Ant navigation task in a ⊃ -shaped maze from Nachum et al.
(2018) in Fig.
6a.
The abstract states of the task are collected on an abstract 2-D gridworld m ∈ R N × N in
Fig.
6b.
An abstract grid ( x, y ) ∈ N × N subsumes concrete positions with a known scale where m [ x, y ]
= 1 indicates walls and m [ x, y ]
= 0 indicates nagivable spaces.
The domain expert can then specify a pair of abstract initial state ϕ
Init (yellow) and goal state ϕ Goal (green).
By abstracting away details of the low-level dynamics such as Ant orientation and velocity, a high-level policy over abstract states and actions can efﬁciently plan over much longer time horizons.
For Ant Push and Ant Fall, we reuse the same state abstraction as above.
We additionally use m [ x, y ] =
Move to specify that there is a movable block at ( x, y ) .
The Ant needs to push away the movable block to reach the goal behind it in Ant Push or push it into a rift and then utilize the block as a bridge to get close to the goal on the other side of the rift.
Other than state abstraction, for model-based planning, we also abstract over actions to ensure a ﬁnite high-level model.
We directly use action abstractions as provided primitive functions therein the ensemble policy DSL because they abstract low-level agent actions to an abstract space of skills.
Our algorithm then leverages abstract action (primitive function) speciﬁcations to synthesize a high-level control plan on the abstract high-level model.
Compared to existing model-based HRL approaches e.g. (Jothimurugan et al., 2021), our algorithm uses abstract actions as speciﬁed rather than learned.
In the following, we show that with taskagnostic abstract actions (primitive functions), providing these speciﬁcations is trivial even for complex high-level planning tasks.
For the Ant Push and Ant Fall environments, the Ant primitives move an Ant up π up , down π down , left π left , and right π right .
Therefore, the abstract actions move an Ant horizontally, vertically, or diagonally (via composition) one step on the abstract model:[-ϕ Act Ant Push ( x i − 1 , y i − 1 , x i , y i , m i − 1 , m i ) ≡ ∃ dx i , dy i . x i = x i − 
 + dx i-]
[-∧ y i = y i − 
-][Equation]
m
[Equation]
1 [ p ][ q ] otherwise
Compared to the ϕ Act deﬁnition in Sec. 4, we add two new parameters
m i − 1 and m i to the new deﬁnition above.
This is because an agent action may change its environment.
Here m i − 1 refers to the old environment before an agent action, and m i refers to the new environment after the agent [Table]
action. For example, for Ant Push, if a movable block is pushed by the Ant, the state abstraction should be updated accordingly reﬂecting the fact that the space after the movable block in m i − 1 holds the block after the push in the new environment m i .
We require that an abstract action be speciﬁed together with a guard ϕ Guard that encodes the condition under which the action can occur.
For Ant Push, the Ant should not walk into walls and the path from an old position to a new position should be available.
Importantly, if in the high-level plan the Ant pushes the movable block
[Table]
there should exist space behind the block that allows for the push in the old environment m i − 1 before the action.
Besides, a movable block can only be moved vertically or horizontally in the environment.
[Table]
[Equation]
[Table]
[-i m i − 
 [ x i ] [ y i ] =-][Equation]
[Table]
[Equation]
[Table]
+
[Equation]
[-[ y i ] = 
 ∨ m i − 1 [ x i ][ y i − 1 ] = 0 (cid:1) ∧ m-] i −[-
-]
[Equation]
[-( y i − y i − 1 )] = 
 ∨ m i − 1 [ x i +-]The abstract action speciﬁcation for Ant Fall can be deﬁned similarly as follows:
[Equation]
Notice that ϕ Act Ant Fall is almost the same as ϕ Act Ant Push except that we specify that if the Ant pushes a movable block into a rift, the rift becomes a bridge for the Ant to bypass. ϕ
[Equation]
Similarly, ϕ Guard Ant Fall is almost the same as ϕ Guard Ant Push except that we additionally specify that the Ant can push a movable block either if there is an empty space or a rift behind the block in the old environment before the action.
G. D ETAILED R ESULTS ON H IERARCHICAL RL B ENCHMARKS
We show the success rates of applying our learning algorithm
(Sec. 4), HIRO (Nachum et al., 2018), and A-AVI (Jothimurugan et al., 2021) in Table 7.
It can be seen that π -HRPL programs signiﬁantly outperform both HIRO and A-AVI.
For example, on the challenging Ant Push environment, our π -HRPL policy achieves 94% success rate of goal reaching far beyond that of HIRO and A-AVI.
H P ROGRAM I NTERPRETABILITY : A NT C ROSS M AZE
In this section, we take one step ahead and show how an Ant Cross Maze program P cross can be interpreted by using both branch activation and also policy strength graphs.
H. B RANCH A CTIVATION
In Sec.2, we demonstrate multiple branch activation graphs given different two targets: (6, -6) and (12, 0), and we analyze how they can be explained.
Furthermore, When the goal is set to (6, 6), in most areas of the maze, branch one of the P cross is activated (shown in Fig. 11a) and the agent would majorly follow the instruction provided by the branch one.
From Fig.3, we can conclude that the semantics of this branch is to navigate the agent to go UP , since the parameter associated with UP primitive policies dominates this branch.
Similarly, Fig.11b shows that int most upper-half of the given area, branch two is activated and it would enforce the agent to go RIGHT .
[Figure]
H. P OLICY S TRENGTH
Unlike a branch activation graph which informs us how a branch is activated, a policy strength graph demonstrates how a speciﬁc primitive skill policy is invoked without considering the semantic of any branch, so it is suitable for ﬁnal analysis and interpretation.
Fig.12a and 12b demonstrate the UP policy strength of program P cross given target locations (12, 0) and (6, -6).
Presumably, if the agent start from origin (0, 0) and the target is (12, 0), it should move straight up and invoke UP policy.
From the Fig.12a we know that the behavior of the program is consistent with our expectation: invoking UP policy throughout its trajectory.
Furthermore, if we change the target position to (6, -6), behavior of how the program invokes UP policy (see Fig.12b) changes accordingly: the strength of UP policy is at a high level in the beginning of the trajectory, then decreases rapidly to prepare for a right turn.
Therefore, the effect of P cross shows in Fig.12 can be summarized as: the strength of UP policy could be adjusted according to different target locations, yet it stays at a high level at the beginning of the trajectory.
The goals shown in Fig.13a and 13b are symmetric.
After reaching the central area of the maze using
UP policy, the program should navigate the ant to make a left or right turn by calling LEFT or RIGHT primitive policies, respectively.
If the goal is located at (6, 6) (see Fig.13a), the LEFT policy strength is high after the agent reaches the center of the maze, so it can be pushed and move towards left.
When the goal is assigned to (6, -6) on the right side of the maze (see Fig.13b), the RIGHT policy is invoked at a high level after the ant reach the center of the maze, so that the agent will be pushed to move right.
Therefore, the effect of P cross shows in Fig.13 can be summarized as: the LEFT or
RIGHT polices strength is at a high level when the ant reaches the center area of maze so it can be navigated to make a left or right turn to reach the targets. [Figure] [Figure]
I P ROGRAM I NTERPRETABILITY : M ORE E XAMPLES
In this section, we demonstrate more examples of programs and their branch activation and policy strength graphs for interpretation.
I. A NT
[Figure]
ANDOM G OAL P ROGRAM
Fig.14 demonstrates the expression of the program P random for solving Ant Random Goal environment.
The pre-deﬁned function input X takes symbolic states include current position x, y along with the randomly sampled target location G x , G y .
Also, a vector of distribution of weights of primitive policies can be found on each terminal branch.
Fig.15 shows the branch activation of the ﬁrst three branches.
The ﬁgures clearly demonstrate that when the goal is set to be ( G x , G y )
=
(-3, -3), roughly only branch one is activated in the needed areas, so the behavior of the agent is dominantly determined by the semantic of branch one.
Since the second and fourth values in the policy distribution vector in branch one nearly equally dominate the branch, the expected behavior of this branch can be summarized as: go RIGHT and go DOWN at the same time.
Fig.16a and 16b demonstrate the DOWN and RIGHT policy strength of a program P random given origin (0, 0) and target location (-3, -3), respectively.
In the ﬁgures, the x-axis and y-axis reﬂect the current position of the agent.
The output of P random is a vector that contains strength of each primitive skill policy and we are interested in the strength DOWN and RIGHT primitive policies in this case.
Presumably, if the goal position is (-3, -3) and the ant starts from (0, 0), the simplest path for an ant is to move diagonally along with southeast direction, and surely this requires DOWN and
RIGHT primitive policies, and each of them is likely to contribute equally.
From Fig.16 we know that with this program P random , when the agent moves southeast, the contribution of DOWN and [Figure] [Figure]
RIGHT are roughly the same, and this is consistent with our expectation.
Therefore, the effect of this program can be summarized as: if the agent is required to move southeast from (0, 0) to (-3, -3), assign DOWN and RIGHT equally at roughly 50 percent of strength.
I. P USHER P ROGRAM
Fig.17 demonstrates a program P pusher with three branches for solving Pusher environment.
The program input X takes symbolic states x obj , y obj , x arm , y arm as inputs, then computes (cid:107) x obj , y obj (cid:107) 2 , (cid:107) x arm , y arm (cid:107) 2 to decide which branch of weight cell to take.
From Fig.18a we can conclude that, only the ﬁrst branch is activated in the whole possible situations.
Such activation values for branch 1 demonstrate that the possibility of cutting other branches in the program without hurting the performance of the graph too much.
Also, it suggests that a smaller value of maximum depth of program derivation graph is sufﬁcient for searching a program to solve this task.
Also, the distribution of primitive policies in this branch shows that: the agent needs more
PUSH-DOWN policy than PUSH-LEFT policy to move the object to the target location.
Furthermore, Fig.18 demonstrates PUSH-DOWN and PUSH-LEFT policy strength of the program
P pusher .
Unlike Ant Cross Maze or Ant Random Goal environment, the goal of pusher environment is at a ﬁxed location.
From Fig.18b we can conclude that: when the object is faraway from the origin, i.e., close to the goal, PUSH-DOWN policy strength is at a relatively lower level, compared to at a higher level when the object is far from the goal position.
Also, we can summarize the effect of how the program invoke PUSH-LEFT primitive policy from Fig.18c: when the object is too close to the origin, the PUSH-LEFT policy strength is at a lower level, and if the object is close to the goal, the
PUSH-LEFT policy strength is at a higher level.
In general, the above behavior of PUSH-LEFT can be explained if we consider velocities in both x and y axes: if we want to acquire a policy which [Figure] [Figure] push the object at a constant tangential velocity all the way to the goal, we may need to adjust the distribution of PUSH-LEFT and PUSH-DOWN in order to get a reasonable combination of velocity in both x and y axes.
I.
[Figure]
HEETAH
[Figure]
URDLE
[Figure] ROGRAM
According to our experiments when applying π - PRL algorithm to HalfCheetah Hurdle environment
[Figure]
the program derivation graph is inclined to select a program with few branches.
In fact, a simple
HalfCheetah Hurdle program P hc with only one branch (shown in Fig.19) is able to solve the task with high performance and success rate.
An example of such program can be written as 95% · π FORWARD ( s ) + 5% · π JUMP ( s ) .
In this simple program, the input X is not even used, this means the agent invokes each primitive policy at a ﬁxed rate.
The success of such a simple combination of two primitive skill policies tells us, if a program is powered by appropriate low-level primitive policies, we may solve a RL task in a much simpler way.
J P OLICY G ENERALIZABILITY
In our experience, the high interpretability of learned programmatic policies leads to strong generalizability of the policies to novel environments.
J. A NT N O M AZE
We consider applying the programmatic policy P cross for Ant Cross Maze to an environment with randomly sampled target positions far beyond the three possible goals in the original environment.
Our goal is to test if the programmatic policy has learned how to perform goal navigation.
Since a new target position can be quite different than the three goals in Ant Cross Maze, we remove all the walls from the environment so the Ant can navigate freely.
Fig.20a and 20b depicts the UP and
[Figure]
[Figure]
[Figure]
RIGHT policy strength if we set the goal to (14, -8), i [Figure] e., the upper-right corner.
Fig.
20c and 20d shows the UP and LEFT policy strength if we set the goal to (14, 8), i [Figure] e., the upper-left corner.
P cross can successfully drive the Ant to these positions.
From these policy strength ﬁgures we can conclude that P cross successfully detects the change of goals and weighs UP and associated RIGHT or LEFT policies in appropriate areas to navigate the ant to reach the goals.
[Figure]
[Table]
[Figure]
Furthermore, we consider a reshaped cross maze with three different goals: (8, 8), (14, 0), (8, -8) (represented by blue spheres in Fig.21).
The shapes and positions of walls have also been changed and adapted to new possible goals accordingly.
This novel environment Ant Reshaped
Maze is similar to Ant Cross Maze environment in many ways, e.g. they both have three possible goals, their goals are symmetric, etc.
We directly apply P cross to Ant Reshaped Maze and compare the generalizability of P cross with the BiLSTM policy learned in the Ant Cross Maze environment as well using
Composition-SAC (Qureshi et al., 2020).
We compare with Composition-SAC because the two policies both learn to compose primtivie skills to adapt to new complex behavior.
However, P cross is more interpretable than a BiLSTM model.
Table 8 shows that P cross successfully realizes the change of the three goals and drive the Ant in the novel environment with 0.16 normalized distance to the new goals.
In constrast, the Composition-SAC policy does not solve the novel environment well.
We further increase the original radius of goals to 1.5 and 2 in the novel environment, both P cross and the Composition-SAC policy get close to goals with P cross continuing to perform much better.
K A FFINE P OLICIES AND PID P OLICIES K. A FFINE P OLICIES
The DSL for afﬁne policies implements a controller C as afﬁne transformations in Fig.
[Equation]
where θ ∈ R m ·|X| , θ c ∈ R m are control parameters.
Particularly, C Afﬁne can be as simple as some (learned) constants θ c .
For example, consider the continuous OpenAI Gym MountainCar environment in Fig. 22a.
A car is on a one-dimensional track, positioned between two “mountains”.
The goal is to drive up the mountain on the right.
Because the car’s engine is not strong enough, a policy has to drive the car back and forth to build up momentum in multiple passes.
The reward is greater if the policy spends less energy to reach the goal.
A learned programmatic policy to control the continuous MountainCar environment is given in Fig.
Fig.
22b depicts the activation of branch 1 in the program as a function of ( position, velocity ) .
The degree of activation (yellow) is close to 1 on almost all states under with speed velocity greater than 0.
The logic of the program is then obvious — when ever the car speed is greater than 0, the policy accelerates the forth to drive the car forward; when ever the car speed is less than 0, the policy accelerates the forth to drive the car backward.
Indeed, this helps drive the car back and forth to build up momentum.
The policy succeeds reaching the goal and receives cumulative reward greater than the threshold 90 on all experienced episodes.
[Figure]
if θ 1 c + θ T 1 · X > 0 then 0 .
92 ←−
Branchelse − 0 .
95 ←− Branch
[Figure]
K. PID P OLICIES
The DSL for PID policies implements a controller C as a discretized, multivariable PID function in
Fig. 1:
[Equation]
where θ P , θ I , θ D ∈ R m · n are parameters representing the proportional gain, integral gain, and derivative gain matrices of PID control. Notice that a PID controller additionally takes a known constant (cid:15) that represents a ﬁxed target for the controller to stabilize the system under control around, and a histroy h of a sequence of states before the current control step.
The semantics of the controller is as follows:
[Equation]
where [Equation] [Equation] In the semantics deﬁnition, P is the proportional term, I is the discrete approximation of the integral term (calculated via a fold ), and D is the ﬁnite-difference approximation of the derivative term.
In line with the standard integral error reset strategy ( ˚Astr¨om & H¨agglund, 1984), the fold function acts over a ﬁxed-sized window on the history (e.g. the ﬁve latest states of the history).
peek ( h, − 1) returns the most resent state in a history h .
For example, consider the OpenAI Gym Pendulum environment in Fig. 24a.
The pendulum swingup problem is a classic problem in the control literature.
The pendulum starts in a random position, and the goal is to swing it up so it stays upright.
The state space include ω that is the angle the pendulum makes with the vertical and the angular velocity ˙ ω .
However, the observation exposed to a policy for this task is [ cos ( ω ) , sin ( ω ) , ˙ ω ] .
Since the goal is to stablise the pendulum upright, the ﬁxed goal (cid:15) for a PID controller for this task is [1 , 0 , 0] (i.e. [ cos ( ω ) = 1 , sin ( ω ) = 0 , ˙ ω = 0] ).
A learned programmatic policy to control the pendulum environment is given in Fig.
Fig.
24b depicts the activation of branch 1 in the program as a function of ( ω, ˙ ω ) .
The degree of activation (yellow) is close to 1 on all states where the PID controller in branch 1 would be applied to.
Accordingly, the large constant control signal in branch 2 would be applied states within the blue [Figure]
if θ 1 c + θ T 1 · X > [Figure] region. The logic of the program is then obvious — when ever the pendulum is hanging or downwards close to + π or − π , the policy applies the large (constant) control action in branch 2 to push the pendulum to an area in which it is easy for the PID controller in branch 1 to eventually stabilise the pendulum upright at 0 , 2 π, or − 2 π .
The policy succeeds on all experienced episodes.
In our experience, we found that it is impossible to train a single PID controller to solve the task.
The programmatic policy in turn uses a large control action to push the pendulum to where the PID controller is able to function.
Experiment Results of Learning Afﬁne and PID Policies.
We have applied the learning algorithm in Sec.
3 to the popular benchmark suite MuJoCo.
This benchmark suite consists of multiple locomotion tasks with 2D and 3D agents.
Table 9 demonstrates that our results on Humanoid-v and LunarLander-v2 are competitive with state-of-the-art RL algorithms (trained using 3 million environment steps). More results were given in Table
[Table] [Table]
a