Deep RePReL–Combining Planning and Deep RL for acting in relational domains
Abstract
We consider the problem of combining symbolic planning and deep reinforcement learning (RL) to achieve the best of both worlds – the generalization ability of the planner with the effective learning ability of deep RL.
To this effect, we extend a previous work of Kokel et al.
[17], RePReL, to deep RL.
As we demonstrate in experiments in two relational worlds, this combined framework enables effective learning, transfer and generalization when compared to the use of an end-to-end deep RL framework
Introduction
The two sequential decision making directions of Reinforcement Learning (RL) and AI planning are complimentary.
While deep RL agents are able to solve complex games such as Go, Atari, etc. [33, 25], they are extremely data hungry (requiring 100 M examples) and limited in ability to generalize to related tasks and domains.
Generalization is essential for the deployment of RL agents.
As an example, an agent trained to stack 5 red blocks as a pyramid should be able to stack 10 blue blocks as a pyramid.
Contrary to the RL approaches, symbolic AI planning methods [12] have focused on developing domain-independent approaches that can be utilized in different domains and are able to scale with varying objects.
However, most AI planning methods rely on complete (symbolic) description of the domain, to guide the search.
This limits the usage of planning approaches in complex domains where it is difﬁcult for humans to articulate the transition functions, the action schema of the domain.
In summary, the symbolic representation of the planners allow for efﬁcient generalization while the Deep RL methods perform efﬁcient and effective learning.
Consequently, various methods have combined RL and AI planning to improve sample efﬁciency and generalization [14, 16, 7, 37, 23, 24, 20, 10].
We aim to highlight the advantages of combining planning with RL and present the capability of one such framework, RePReL [17], to learn deep RL agents.
RePReL framework uses a relational hierarchical planner at the higher-level to decompose a task into sub-tasks (options), and learns RL policies for options at the lower level, while using task-speciﬁc state representations.
This hierarchical framework provides two key advantages.
First, the hierarchical planner effectively captures the symbolic, higher-level task knowledge, which is more general than simply providing intrinsic rewards.
Second, learning separate policy for each option at lower level allows for task-speciﬁc abstract state representations.
By incorporating deep RL agent in the RePReL framework, we provide a third advantage.
We empower lower level options in RePReL to learn complex functions with ability to act in continuous state and action spaces.
In the proposed deep ReRPeL framework, the planner serves as the higher level, symbolic, deliberate slow reasoner while the deep RL is the fast, effective neural learner, thus achieving the two-level neurosymbolic system, a key goal of many AI/ML researchers.
We focus on relational domains, where the number of entities and relations are not ﬁxed in advance.
To learn to act in such relational domains, it is important for the agent to identify relevant entities for the given task and make decisions by focusing only on the applicable properties and relations of these entities.
For example, while learning to stack all the blocks in a single tower, the agent should not over-ﬁt to the color of the blocks.
In RePReL framework, we leverage the domain knowledge of humans to infer the relevant properties using ﬁrst-order conditional inﬂuence statements.
In this work-in-progress report, we share our initial results on two domains, extended relational taxi domain and ofﬁce world.
We show the capability of the deep RePReL framework to generalize over different objects in the domain and transfer to different tasks.
Proposed Approach
We consider the problem of learning to execute multiple tasks in relational domains.
Thus, we extend the relational MDP (RMDP) deﬁnition from Fern et al.
[9] for goal-oriented domains.
Deﬁnition 1.
A goal-directed relational MDP (GRMDP)
M is represented by (cid:104) S, A, P, R, γ, G (cid:105) , where the states S and the actions A are represented by a set of objects E , a set of predicates Q , and action types Y .
P is a transition probability function S ×
A × S → [0 , 1] , R is a reward function
S ×
A × S → R , γ ∈ [0 , 1) is the discount factor, and G is set of goals that the agent may be asked to achieve.
Different tasks can be formulated by choosing different goals from the potentially inﬁnite set G .
The reward function R provides the reward (or cost) of taking a step in the environment, regardless of the goal.
A separate intrinsic reward t R is used when the terminal state for any goal or subgoal is reached.
A problem instance for a GRMDP is deﬁned by a pair (cid:104) s ∈ S, g ∈ G (cid:105) , where s is the initial state and g is the set of goal conditions, both represented using sets of permitted literals, i.e. positive and/or negative atoms.
A solution is a policy that starts from s and ends in a state satisfying g with probability 1 . 0 .
The RePReL framework solves the GRMDP using a combination of planning and RL in 3 stages, as shown in Figure 1.
1 Planning: Use a hierarchical planner to decompose the goal of the GRMDP into a sequence of subgoals using decomposition rules or methods.
These high-level subgoals act as options.
2 Abstraction: Use human’s task knowledge to abstract the state representations by ﬁltering the irrelevant entities, properties and relationships.
This provides sub-task speciﬁc state representations.
3 RL: Learn a separate RL policy for each option in the abstract state space that achieves the subgoal.
We refer to Kokel et al.
[17] for details of the hierarchical planner.
Here it sufﬁces to say that given a state s and a goal g , the hierarchical planner provides a high-level plan Π =
[ o 1 , o 2 , ..., o n ] which is a sequence of options (or operators).
Each option o is deﬁned with initiation set
I ( o ) (preconditions of the operator) and termination condition β ( o ) (necessary effects of the operator).
A subgoal RMDP M o is deﬁned for each option o , which is solved to obtain the option policy π o .
Deﬁnition 2.
The subgoal RMDP M o for an option o is deﬁned as a tuple (cid:104) S, A, P o , R o , γ (cid:105) consisting of states S , actions A , transition function P o , reward function R o , and discount factor γ .
State and action spaces remain same as the original RMDP.
The reward function R o and transition probability distribution function P o are deﬁned as follows:
R o ( s, a, s (cid:48) )
= (cid:40) t R + R
( s, a, s (cid:48) ) if s (cid:48)
∈ β ( o ) and s / ∈ β
( o ) if s (cid:48) ∈ β ( o ) and s ∈ β ( o )
R ( s, a, s (cid:48) )
otherwise
P o ( s, a, s (cid:48) ) = (cid:40)if s ∈ β o ) and s (cid:48) /
∈ β ( o ) if s ∈ β ( o ) and s (cid:48) ∈ β ( o ) P ( s, a, s (cid:48) ) otherwise with a ﬁxed terminal reward t R and the reward function R ( s, a, s (cid:48) ) from the GRMDP.
Essentially, the reward function in the original GRMDP would correspond to the step cost function, which applies to all options, and reward R o is the only goal-speciﬁc reward.
Next, we present how we obtain the safe state abstraction using the bisimulation framework of Givan et al.
[13] and Ravindran and Barto [30], which has been called “model agnostic abstraction” in Li et al.
Deﬁnition 3 (Li et al.
An abstraction φ ( s ) is model-agnostic when φ ( s 1 ) =
φ ( s 2 ) if and only if for any action a and an abstract state s ,
[Equation]
The ﬁrst condition above states that the two states s 1 and s 2 have the same immediate reward distribution with respect to the abstraction.
The second condition indicates that these two states have the same transition dynamics.
It is proven that Q-learning with such abstraction results in an optimal policy for ground MDP [21].
Since states are conjunctions of literals in RMDPs, to get a safe abstraction, we need to infer which predicates inﬂuence the rewards and the goals of options.
We capture this knowledge using First-Order
Conditional Inﬂuence (FOCI) statements [26], one of the many variants of statistical relational learning languages [11, 29].
Each FOCI statement is of the form: “if condition then X 1 inﬂuence X 2 ”, where, condition and X 1 are sets of ﬁrst-order literals and X 2 is a single literal.
It encodes that literal X 2 is inﬂuenced only by the literals in X 1 when the stated condition is satisﬁed.
For RePReL, we simplify the syntax and extend FOCI to dynamic FOCI (D-FOCI) statements.
In addition to direct inﬂuences in the same time step, D-FOCI statements also describe the direct inﬂuences from the literals in the current time step to the literals in the next time step.
To distinguish the two kinds of inﬂuences, we add +1 on the arrow between the sets of literals to capture a temporal interaction, as option : { p ( X 1 ) , q ( X 1 ) } +1 −→ q ( X 1 ) .
It says that, for the given option , the literal q ( X 1 ) in the next time step is directly inﬂuenced only by the literals { p ( X 1 ) , q ( X 1 ) } .
Following the standard DBN representation of the MDP, we allow action variables and the reward variables in the two sets of literals.
option may be skipped to represent unconditional inﬂuences.
The D-FOCI statements can be viewed as a relational versions of the dynamic Bayesian networks (DBNs) and have a similar function of capturing the conditional independence relationships between domain predicates at different time steps [18].
For example, the D-FOCI statement in the taxi domain for pickup option can be expressed as, pickup(P) : { taxi-at(L1) , at(P, L2) } +1 −→ in-taxi(P) .
This denotes that while the task pickup(P) is being performed, only the taxi location and the passenger location, taxi-at(L1) and at(P,L2) , inﬂuence in-taxi(P) .
This means that the in-taxi(P) is independent of dest(P,D) and at-dest(P) .
Note that contrary to this, when the passenger is being dropped, the dest(P,D) will inﬂuence in-taxi(P) .
Using the substitution θ of the grounded option determined by the planner, the D-FOCI statements would be partially grounded.
For example, using the θ = { X/p 1 , L/r } for pickup(X) the above D-FOCI would become pickup ( p 1) : { taxi-at(L1) , at ( p 1 , r ) } +1 −→ in-taxi ( p 1) .
Hence, only the pickup location of passenger p 1 (i.e., at ( p 1 , r ) ) and the current taxi location taxi-at(L1) (i.e., the current location L1 ) are relevant for grounded operator pickup ( p 1) .
If there is another passenger, say p 2 , then the state variable at ( p 2 , · ) would not be relevant.
While the planner works in relational representations, the reinforcement learning operates at a propositional level.
The gap is bridged by computing an appropriate propositional abstraction of the state for each option with the parameters, e.g., p 1 , bound to generic objects (Skolem constants in logic).
To keep the size of the propositional representation bounded, we bound the depth of inference chain through D-FOCI statements to k .
If the MDP satisﬁes the D-FOCI statements with a ﬁxed depth unrolling, then the corresponding model-agnostic abstraction has the same optimal value function as the fully instantiated MDP.
We defer to Kokel et al.
[17] for the proof and unrolling process.
We present the learning procedure next.
Given the GRMDP environment env , the planner P and the D-FOCI statements F , we now discuss the deep RePReL learning procedure from Algorithm 1 .
First for each option, an RL policy π o and a replay buffer D O is initialized in line 1, 2 .
Next, for the current episode a high level plan Π is obtained from the planner P in line 6 . We employ the SHOP planner Nau et al.
For every grounded option in the plan, we collect the training sample in buffer D
o ( lines 7–23 ).
To collect samples, we get an abstract propositional state representation ˆ s .
In lines 11–22 , we obtain action a from the current policy, perform that action, observe the next state s (cid:48) and the reward r .
If s (cid:48) is a terminal state for the ground option o g , then we add a terminal reward t R ( line 18 ) before adding it to the buffer ( line 20 ).
Then for each option o , the option policy π o is updated after sampling a batch from buffer D o ( lines 25–28 ). [Table]
for episode ∈ k do 5: s ← get state from env 6: Π ← P ( s, g ) (cid:46) get high-level plan 7: for o g in Π do 8: π ← π (cid:15)o (cid:46) get resp.
RL policy 9: ˆ s ← GetAbstractState
( s, o g , F ) 10: done ← ˆ s ∈ β ( o g ) (cid:46) check terminal state 11: while not done do 12: a ← π (ˆ s ) (cid:46) get action 13: s (cid:48) ← env.step ( a ) (cid:46) take step in env 14: r ← R ( s, a, s (cid:48) )
(cid:46) get step reward 15: ˆ s (cid:48) ← GetAbstractState ( s, o g , F ) 16: done ← ˆ s (cid:48)
∈ β ( o g ) (cid:46) check terminal next state 17: if done then 18: r = r + t R (cid:46) add terminal reward 19: end if 20:
D o ← D
o ∪ { ˆ s, a, r, ˆ s (cid:48) , o g } 21: s, ˆ s ← s (cid:48) , ˆ s (cid:48) 22: end while 23: end for 24: end for 25: for each option o ∈ O do (cid:46) Update all the parameters 26:
D ← SampleBatch
( D o , b ) (cid:46) Sample a batch from corresponding buffer 27: π o ← UpdatePolicy( π o , D ) 28: end for 29: end for 30: return π o , ∀ o ∈ O
Experiments
We now present our initial results in two domains, Taxi and Ofﬁce World.
We design our experiments to explicitly answer the followings questions.
Q1:
Sample Efﬁciency: Do the abstractions induced in RePReL improve sample efﬁciency?
Q2:
Transfer: Do these abstractions allow for effective transfer across tasks?
Q3:
Generalization: Does RePReL efﬁciently generalize to varying number of objects? R
G B Y Domains: As mentioned earlier, we evaluate our framework on two domains: an extended relational version of the Taxi domain [5] and an Ofﬁce World [16].
Extended Taxi domain is an 8 × 8 grid, shown in Figure 2, with one taxi and 4 special locations: (cid:104) R, G, B, Y (cid:105) .
There can be more than one passenger at a time in the grid; but taxi can be hired only by a single passenger.
In every episode, the pickup location and the drop location of each passenger are sampled from the 4 special locations.
The agent can perform 6 actions: up, down, right, left, pick, drop .
Environment provides a reward of − 0 .
1 for every step and − 1 for pick or drop action in wrong locations.
We consider the following three tasks in this domain: task 1 , drop a passenger to its destination; task 2 , drop two passengers; and task 3 , drop three passengers to their destination location.
Ofﬁce World is a 9 × 12 grid, shown in Figure 3.
It has one ofﬁce location (indicated with hand), two coffee locations (indicated by mugs), one mail room (indicated by envelope), and some plants in the ofﬁce to be avoided (indicated by ∗ ).
The agent can perform 4 actions: up, down, right, and left .
We consider two tasks in this domain: task 1 , deliver coffee to ofﬁce (which can be picked up by visiting any coffee location) and task 2 , deliver mail to ofﬁce.
Environment provides a cost of 1 at every step and cost of 10 on taking invalid actions like stepping on the plant or moving in the walls.
Baselines We evaluate our deep RePReL agent against a Deep Q-Network (DQN) , an Hierarchical DQN (HDQN) , and a Taskable-RL (TRL) agent.
Mnih et al.
[25] proposed DQN that uses Q updates to train a deep RL agent.
HDQN Kulkarni et al.
[19], integrates the temporal abstraction with intrinsic critic by using two-level controller.
At the higher level, a meta-controller selects the temporally extended actions, also called options , and receives the reward from the environment.
While, the lower level controller learns to act in the environment to achieve the option and receives the intrinsic reward from the critic.
TRL [16] combines symbolic planning with the RL, where the symbolic planner provides the higher-level options and several RL agents are learned for each option at lower level.
The DQN agent learn a single end-to-end policy for achieving the task; while others decompose the task into subtask and learn policy for achieving subtasks.
Further, the HDQN agent learns a meta-controller to generate a high-level plan; while the TRL and the RePReL agent leverage a planner to provide the high-level plan.
And ﬁnally, the TRL agent uses same state representation to learn each policy; while the RePReL uses D-FOCI statements to obtain the task-speciﬁc representation while learning the subtask policies.
All RL algorithms were implemented on a fork of RL-kit 1 .
Table 1 in the Appendix provides details on the hyperparameters used for training all the agents.
All the results presented below are aggregated over 10 runs.
Sample Efﬁciency: To evaluate the effect of RePReL abstractions, we compare it against the seq variant of the Taskable RL.
We pick this variant for two reasons: 1. seq variant performed best in all their experiments, 2.
We aim to evaluate the effectiveness of abstractions and thus do not learn the meta-controller introduced by the partially ordered plans.
We set a budget of 1 e 6 on the number of steps that can be taken in training environment for learning in each task.
Figures 4 (a–e) compares the learning curves of the four agents on all ﬁve tasks.
While the RePReL, TRL and DQN agent achieves the optimal reward after 40 K steps in Taxi task 1 ; their performance signiﬁcantly differs in Taxi task 2 and Taxi task 3 where the number of passenger increases.
In two tasks of Ofﬁce World also we see that RePReL achieves the optimal reward faster than others.
Hence, we answer Q1 afﬁrmatively in that RePReL abstractions statistically signiﬁcantly outperforms the state-of-the-art hybrid planner-RL architecture, Taskable RL, as well as approaches that only learn from data, HDQN and DQN. [Figure]
Transfer: We next evaluate the agents ability to transfer to a related task in the same domain.
To this effect, we transferred all the agents learned for performing the task 1 and the task 2 in Ofﬁce World to perform the task 2 and task 1 , respectively.
We swap the task of the agents after they are trained for 1 e 6 steps.
Performance of the transferred agents are presented in Figures 4 (e–f).
All agents start with a higher average reward (than learning from scratch as seen in Fig.
4 (d–e) ), but RePReL has the steepest learning curve and achieves optimal reward very fast.
This allows us to answer Q2 afﬁrmatively.
Generalization: For evaluating generalization, we transfer all the agents trained on Taxi task 1 to task 2 and train them on task 2 .
Subsequently, we transfer agents from task 2 to task 3 .
Figures 4 (h–i) presents the learning curve of these transferred agents.
While the transferred TRL, HDQN and DQN agents have steeper learning curves than the respective agent learning from scratch (in Fig.
4 (b–c)), the transferred RePReL agent shows zero-shot generalization ability on both tasks.
These transfer results in allows us to answer Q3 afﬁrmatively in that RePReL allows for successful generalization across varying number of objects and is best suited for relational domains.
Discussion and Related Work
State abstractions has been shown to be useful for transfer learning in the literature [36, 34, 1].
Our approach on task-speciﬁc state abstraction is inspired from Dietterich [5] and the bisimulation conditions [30, 13] to deﬁne abstractions in relational settings using ﬁrst-order probabilistic models [29].
Relational Reinforcement Learning (RRL) is another set of works that is related to ours.
RRL aims to learn a policy in a relational world in the presence of multiple objects.
RRL literature [35, 28, 31, 4, 15] tried learning a value function or a policy on a relational representation of the state by using relational induction, symbolic dynamic programming and linear programming alternative but these methods have failed to adapt to mildly complex environments.
With the advent of NeuroSymbolic Theorem provers and Neural Logic Reasoner [8, 6, 32, 3] there is a renewed interest in the community to attempt the relational domains.
While our initial evaluation of RePReL is quite encouraging, there are several avenues that need more evaluation.
First is using the Graph Neural Network (GNN) [2] based architecture to learn the policy.
Few recent work using Graph-based neural network have shown tremendous improvement over the traditional Neural Network based architecture for relational domains.
Speciﬁcally, Zambaldi et al.
[38] introduce the relational inductive biases in the deep learning using GNNs, claiming that such architectures can perform reasoning, and show zero-shot transfer to complex domains.
Inspired by this, Li et al.
[22] use GNN architecture for multi-object manipulation and show that a curriculum learning approach can be order of magnitude more sample efﬁcient than the traditional architecture.
Hence it is imperative we compare our approach with a GNN based architecture.
While this is our current research direction, we emphasize that our proposed RePReL framework is generalizable to any network architecture used for the policy, thus noting the wide applicability of this combination.