Generalised Task Planning with First-Order Function Approximation
Abstract: Real world robotics often operates in uncertain and dynamic environments where generalisation over different scenarios is of practical interest.
In the absence of a model, value-based reinforcement learning can be used to learn a goal-directed policy.
Typically, the interaction between robots and the objects in the environment exhibit a ﬁrst-order structure.
We introduce ﬁrst-order, or relational, features to represent an approximation of the Q-function so that it can induce a generalised policy.
Empirical results for a service robot domain show that our online relational reinforcement learning method is scalable to large scale problems and enables transfer learning between different problems and simulation environments with dissimilar transition dynamics.
Introduction
Adaptation to different scenarios is essential for real world robotic applications.
As a motivating example, consider a service robot interacting with people and objects as shown in Figure 1.
The robot is required to attend to a person who requires assistance and complete the tasks given by the person.
The uncertainty in who needs assistance, what these task could be, and the number of people and objects in the environment necessitates generalisation of learned knowledge from past observations in order to complete new but similar tasks in a different environment.
If the model of the interaction between the robot and its environment is unknown due to its complexity, then value-based reinforcement learning (RL) methods can be used to solve the planning problems.
The amount of observations required to achieve near-optimal behaviour, or sample complexity, is usually prohibitive, especially in large scale problems.
This restricts the practicality of RL methods in applications where data collection is expensive.
One solution is to utilise transfer learning which can reduce sample complexity by leveraging the knowledge learned in small scale problems where learning can be more efﬁcient, or in simulated environments where data collection is cheap and safe.
Interactions between robots and objects in its environment often exhibit a ﬁrst-order structure (i.e., ground actions of a lifted action change the relations or properties of all objects of the same type in similar manner).
We model problems with a ﬁrst-order structure using Relational Markov Decision Processes (RMDPs).
Relational reinforcement learning (RRL) learns in a ﬁrst-order representation rather than a propositional representation and can achieve knowledge transfer without the need of a mapping between problems [1].
In this work, we propose a model-free RRL method where the action-value function, or Q-function, is approximated by projecting the state space into a lower dimensional space using a set of features.
We utilise an existing online feature discovery algorithm, iFDD+ [2], to incrementally add conjunctive state predicates as features to reduce the approximation error.
Since state predicates are ground over objects, problems with different objects or numbers of objects have different sets of state predicates, or feature spaces, and generalisation is not possible.
Instead, we use lifted state predicates, or ﬁrst-order features, to approximate the Q-values of lifted actions rather than ground actions.
By learning in a abstract state-action space which is common across all problems of the same domain ( related problems ), knowledge transfer is possible.
Our contribution is threefold.
First, we propose a method to generate a ﬁrst-order feature space automatically given a RMDP without the use of a model, expert knowledge, or training data.
Second, we implement an online, model-free RRL method which learns a ﬁrst-order linear function approximation of the Q-function given an initial set of ﬁrst-order features.
The approximated Q-function induces a generalised policy which allows transfer learning between related problems independent of the objects, number of objects, initial states, and goal states.
Third, we introduce the concept of contextual knowledge to reduce granularity in the ﬁrst-order function approximation, improve plan optimality, and reduce computational cost.
We evaluate our method empirically on randomised problems in a service robot domain and in different simulation environments.
Related Work
Exploitation of the ﬁrst-order structure of planning problems could reduce the sample complexity of
RL methods.
There are broadly two types of methods: model-based and model-free.
Model-based methods such as [3, 4, 5, 6] learn relational models and employ planning techniques.
Our work is a model-free RRL method and, therefore, we focus on reviewing model-free learning methods.
Mausam and Weld [7] and D ˇ zeroski et al.
[8] partition the state space into ﬁner regions, each of which has a real-value representing the Q-value, using relational regression trees.
Wu and Givan [9] perform supervised learning of relational features, represented as decision trees, by adding features which correlate well to the Bellman error of value functions.
A decision tree is sensitive to the order of node splitting and is ill-suited in online RL where later observations yield new information which might necessitate the reconstruction of the trees.
Ramon et al.
[10] propose a tree restructuring operation but require statistics to be stored for every node.
We use conjunctive features similar to SVRRL [11] which avoids the issue faced by decision trees.
SVRRL represents a value function with a relational naive Bayes net and learns both the values and structure of the network.
It utilises a distance metric to generalise over handcrafted non-binary relational features.
Likewise, RIB [12], which uses instance-based learning where selected observed examples are stored, requires a domainspeciﬁc distance metric to compute Q-values of unseen state-action pairs.
Our work does not require any distance metric to be deﬁned.
In these aspects, [13] and [14] are most similar to our work.
The former samples features from a set of features with the use of prior training data while we discover features online.
The latter restricts features to have at most one free variable while our features can have any number of free variables.
Other works such as [8, 10, 15, 12] do not consider free variables.
Instead, only objects in the arguments of the goal predicate and action are considered which restricts the applicability of their methods to problems with simple goals.
Some approaches require extensive domain knowledge to deﬁne speciﬁc representations that facilitate efﬁcient learning.
Morales [16] deﬁnes abstract actions (r-actions) and abstract states (r-states) in a relational representation, then learns policies in this abstract state-action space.
Konidaris and Barto [17] learn a shaping reward in the action space, represented by sensory measurements which are common and hold the same semantics for all problems, to accelerate learning in another problem.
In later work [18], options in the agent-space are learned and transferred.
Van Otterlo [19] solves an abstract MDP, which is created with the use of background knowledge, of the underlying
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table]
RMDP.
Sharma et al.
[20] perform case-based learning and quantify state similarity by the Euclidean distance between a pair of states which are represented by handcoded, real-valued state variables.
The Q-values of an action are a weighted sum of the Q-values of matching cases, weighted by how similar these cases are to the current state.
Our work uses trivial domain knowledge for contextual grounding which we posit are known in most, if not all, planning problems anyway.
Preliminaries
We solve planning problems represented with RMDPs using RL methods where the Q-function is approximated with a linear function approximation.
Relational Markov Decision Process (RMDP).
Factored Markov Decision Processes (MDPs) [21] represent a state s with a set of state predicates P where s = (cid:86) | P | i =1 p i and p i ∈ P .
We use symbols with boldface to indicate sets.
If the transition of p i depends only on a small number of state predicates ¯ P ⊂ P , then the transition function T , which deﬁnes a probability distribution over possible successor states after executing an action, can be represented compactly; for a predicate p
i , T ( p (cid:48) i | P , a ) =
T ( p (cid:48) i | ¯ P , a ) where p (cid:48) i is the predicate at the next time step and a is an action.
A RMDP is a ﬁrst-order representation of a factored MDP given by the tuple ( O , P , S , A , T , R , s 0 , H, γ ) .
O is a set of objects, each associated with a type, P is deﬁned over O , S is a set of all possible state speciﬁcations over O and P , A is the set of all possible instantiated actions, T : S ×
A × S → [0 , 1] , R : S ×
A → R is the reward function, s 0 is the initial state, H is the planning horizon, and γ is the discount factor.
Each predicate is applied over a type-consistent tuple of objects.
Different planning problems of a domain can be constructed where their RMDPs have different O , initial states, and goal states.
A generalised policy is a policy π : s → a which directly solves any problem of a domain.
Linear Function Approximation.
The Q-function estimates the expected return, or Q-value
Q ( s, a ) , of executing an action a in the state s and following a policy thereafter.
The Q-function can be represented with some form of parametric function approximation such as a linear combination of features:
(cid:101) Q = (cid:80)
ki =1 w i φ
i , where Φ = { φ 1 , . .
. , φ k } is a set of features and w = { w 1 , . .
, w k } is a set of scalar weights.
A feature φ i maps ( s, a ) to a scalar.
The learning objective is to ﬁnd the set of features and weights such that (cid:101) Q closely approximates the optimal Q-function.
A policy π induced by the optimal Q-function maximises the total discounted reward (cid:80) Ht
=
γ t r t , where r t is the reward observed at time step t .
A feature partitions the state space into regions such that states in a region are considered the same for the purpose of approximating the Q-values.
The coverage of a feature is the region of the state space for which the feature evaluates to non-zero [22].
Features with low coverage give ﬁne granularity in approximation and thus have better accuracy than features with high coverage.
This impacts the soundness of policies.
Conversely, features with high coverage offer better generalisation as learning is done over a smaller number of partitions of the state space.
Online Learning with First-Order Linear Function Approximation
Our objective is to learn a generalised policy which can be used to directly solve every problem of the same domain.
We present our online RRL method in Algorithm 1 which learns a linear function approximation of the Q-function, (cid:101) Q , where the features Φ are ﬁrst-order (initialised in line 1). Algorithm 1 uses iFDD+ [2] for online feature discovery.
(cid:101) Q is deﬁned over the abstract state space (represented by the ﬁrst-order features) and lifted action space which are the same in all related problems.
This allows transfer learning where (cid:101) Q is learned in a problem and transferred directly to solve another problem.
Thus, the policy π (cid:101) Q induced by (cid:101) Q is a generalised policy.
At each time step t (line 2), the robot executes an action a t given by π (cid:101) Q (lines 3 and 4) and observes the successor state s t +1 , reward r t , and duration of execution ∆ t .
The temporal difference (TD) error is computed and the weights w are updated accordingly (line 5) with a TD learning method.
We used Double Q-learning [23] with replacing eligibility traces for each feature [24].
iFDD+ adds new features, which are the conjunction of two existing features in Φ , to Φ if their relevances η t exceed the discovery threshold ξ (line 6).
This reduces the approximation errors of (cid:101) Q by introducing a ﬁner granularity in (cid:101) Q .
In the remainder of this section, we elaborate on Algorithm 1.
4.1 Generalisation with First-Order Features
A feature φ i is a set of conjunctive state predicates or a single state predicate which evaluates to 1 if it is satisﬁed in a state; otherwise it evaluates to 0 .
We denote a function approximation which uses ground state predicates for features as a ground approximation .
Let (cid:101) Q 1 be a ground approximation for a RMDP with state-action space S 1 ×
A 1 , and φ i be one of its features.
Evaluating the value of φ i = p 1 ∧ . . .
∧ p n ∈ S 1 is straightforward.
However, in a second RMDP (i.e., a different planning problem) with state-action space S 2 × A 2 , φ i cannot be evaluated if any of its constituent state predicates p 1 , ..., p n is not in S 2 .
Extending this further, (cid:101) Q 1 ( s, a ) does not map to a real value if s ∈ { S 2 − S 1 } or a ∈ { A 2 − A 1 } .
Thus, (cid:101) Q 1 cannot induce a policy for the second RMDP unless S 1 = S 2 and A
=
A 2 .
However, the two RMDPs are trivially identical with possibly different initial states and goal states.
We are interested in transfer learning between related problems with different state-action spaces.
This motivates our use of a ﬁrst-order approximation which is independent of S , A , and O .
In a ground approximation, the features for each ground action a , Φ a , are initialised as the set of every state predicate and their negation: Φ :=
[ Φ a 1 , ..., Φ a m ] , where m is the number of ground actions.
We use ˆ to denote a lifted predicate or a set of lifted predicates and lowercase (uppercase) letters to represent variables (objects).
In a ﬁrst-order approximation, we instantiate a set of ﬁrst-order features for each lifted action ˆ a instead of every ground action.
Otherwise, (cid:101) Q will not be independent of O because ground actions are deﬁned over O .
ˆ a can be ground with the variable binding σ a = { x/X, y/Y } to a ground action a ( X, Y ) .
For every ˆ a ∈ (cid:98) A , we initialise Φ ˆ a (line 1 of Algorithm 1) with the following steps:
We denote the set of ground actions for ˆ a as A ˆ a .
The initial set of ground features for every ground action a ∈
A ˆ a are lifted in accordance with the binding σ a where objects in the arguments of a are substituted with their corresponding bound variables .
Φ ˆ a is the union of the features from step 1.
The remaining objects are substituted with free variables , where (cid:63)obj denotes a free variable of type obj .
This procedure will generate the same set of ﬁrst-order features for every related problem.
Since (cid:98) A is also the same, our ﬁrst-order approximation has the same representation in all related problems.
A ﬁrst-order feature must be fully grounded in order to evaluate its value in a state.
In contrast to bound variables which are ground with objects in the arguments of actions, free variables can be ground to any object of the same type.
We discuss the grounding of free variables in Section 4.3.
Service robot domain.
We introduce a novel domain, Service Robot ( SR ), which is illustrated in a Gazebo environment in Figure 1.
In SR , a TIAGo robot has to assist some people.
The robot navigates from location wp 1 to wp 2 with the action move ( wp 1 , wp 2 ) .
A person can request assistance (cid:0) represented by the state predicate need assistance ( person ) (cid:1) which is a probabilistic exogenous effect (i.e., it is not due to the robot’s actions).
The robot does not initially know where the people are located and has to ﬁnd them (cid:0) find ( person ) (cid:1) .
It then moves to the person and talks to them (cid:0) talk ( person ) (cid:1) .
If the person has requested assistance, the robot would receive some tasks (or goals) of two possible types: (1) bring an item obj to location wp (cid:0) goal object at ( obj, wp ) (cid:1) , or (2) deliver obj to person (cid:0) goal object with ( obj, person ) (cid:1) . The robot could pick up an item (cid:0) pick up ( obj, wp ) (cid:1) , put down an item (cid:0) put down ( obj, wp ) (cid:1) , give an item to a person (cid:0) give ( obj, person ) (cid:1) , or take an item from a person (cid:0) take ( obj, person ) (cid:1) .
Example 1.
In a ground approximation, we use the predicates ¯ P = { need assistance ( P 1 ) , ..., need assistance ( P m ) } as features (i.e., | ¯ P | number of features), where P i are objects of type person .
In a ﬁrst-order approximation, these predicates are lifted to a ﬁrst-order feature.
For the lifted action talk ( person ) , φ i = need assistance ( person ) .
The number of ﬁrst-order features does not increase with the number of person objects unlike the ground approximation.
This demonstrates that the ﬁrst-order approximation does not scale with the number of objects and is independent of O .
To evaluate φ i , it must be grounded.
The bound variable person is substituted according to the variable binding of talk (e.g., for talk ( P 3) , person is substituted with P 3 ).
One of the ﬁrst-order features for move ( wp 1 , wp 2 ) is φ j = need assistance ( (cid:63)person ) ; because move does not have an argument of type person , P i is substituted with a free variable (cid:63)person .
φ j could be grounded by substituting (cid:63)person to any object of type person .
4.2 Online Feature Discovery
In Section 4.1, we describe the initialisation of Φ where each feature is a lifted state predicate.
There could be large approximation errors in (cid:101) Q as each of these feature has a high coverage.
Hence, adding conjunctive state predicates, which have lower coverage, to Φ could reduce the approximation error.
The problem of feature discovery is to determine which features to add.
We utilise iFDD+ , a modelfree, online feature discovery algorithm (line 6 of Algorithm 1).
A set of candidate features Φ ca consists of features which are the conjunction of any two features (its parent features ) in Φ a .
We denote the generation of Φ ca from Φ a as P ( Φ a ) .
iFDD+ iteratively adds candidate features to Φ a at each time step if their cumulative absolute approximation errors, or relevances, exceed the discovery threshold ξ which is a user-deﬁned parameter.
The relevance of a candidate feature is updated at each time step by accumulating the TD error only if the candidate feature is active.
A feature is active if it is true in the state and is not a parent feature of any active feature.
The weight of a newly added feature is equals to the sum of the weights of its parents.
When a feature φ i is added to Φ a , this in turn generates new candidate features with φ i as one of its parent features.
4.3 Contextual Grounding of Free Variables
To evaluate the values of ﬁrst-order features in a state, they must be fully ground.
This is required in
Algorithm 1 to determine the Q-values (lines 3 and 5) and the active features and candidate features (line 6).
The grounding inﬂuences the step update of w (line 5) and feature discovery (line 6).
Bound variables are substituted with objects of the same type in the arguments of a ground action (see Example 1).
Features with at least one free variable are not fully ground.
The set of possible substitutions for these features consists of every combination of objects for the free variables.
The number of substitutions for a ﬁrst-order feature φ i and for Φ ˆ a are:
[Equation]
respectively, where σ φ i ( σ Φ ˆ a ) is the set of possible substitutions for φ i ( Φ ˆ a ), v is the set of free variables in φ i and | v | is the number of objects of the same type as v .
If we consider every substitution (i.e., to treat a free variable like an existential quantiﬁer) where φ i is true in s if there exists a substitution in σ φ i which makes it true, then there would be many such states (e.g., in Example 1, need assistance ( (cid:63)person ) will be true if any person needs assistance).
Thus, ﬁrst-order approximation has a coarse granularity which can deteriorate performance.
We propose two methods to resolve this.
First, we use contextual knowledge to reduce the number of objects which a free variable can be grounded to (i.e., reduce | v | in Equation 1).
Any substitution which conﬂicts with the substitution due to contextual knowledge is removed from σ Φ ˆ a .
Two substitutions conﬂict if they substitute a free variable with a different object.
We introduce two forms of contextual knowledge which require trivial domain knowledge: goal and location.
Second, after contextual knowledge is applied, we select σ M ⊆ σ Φ ˆ a such that a metric M is maximised: σ M = { arg max σ ∈ σ Φ ˆ a M (cid:0) Φ σ ˆ a (cid:1) } , where Φ σ ˆ a is the grounding of Φ ˆ a with σ .
We deﬁne M (cid:0) Φ σ ˆ a (cid:1) =
(cid:80) φ i ∈ Φ σ ˆ
a φ
i ( s, a )( | φ i | ) 2 , where | φ i | is the number of state predicates in φ i .
This maximises the number of active, complex features.
The choice of M must be dependent on the state only to preserve the Markovian property of RMDPs.
Goal Context.
The goals G in a planning problem are state predicates which can be trivially determined from the deﬁnition of terminal states which are commonly assumed to be known.
The objects in g ∈ G are used for grounding free variables of the same type: σ goal =
g = { (cid:63)x 1 /X 1 , . .
, (cid:63)x n /X n } where g = p ( X 1 , . . .
, X n ) .
We consider only active goals: a goal is active if it is known and has not been achieved (e.g., in SR , a goal is known only after a person has given a task).
We assume that achieved goals are inconsequential for decision making in the present and future.
The Q-values after applying goal context represent the expected values of actions for achieving the goal.
This is similar to the goal-associated Q-function in [25] but is not the same as additive rewards as deﬁned in [26] where each goal is assumed to contribute uniformly and additively to the reward.
φ i =
if there exists a g such that σ goal = g results in φ i being true; the value does not change even if more of such goals exist.
A goal typically gives a contextual grounding which is in conﬂict with another goal.
We propose two goal selection schemes: (1) GA uses every goal with the maximal metric M , and (2) G uses only one goal at a time.
G1 selects the next goal in an ordering of active goals, ¯ G , when the currently selected goal is achieved.
The order in ¯ G could affect the optimality and soundness of the policy.
We determine ¯ G from the state trajectories observed in the previous episode.
A partial order g i <
g j is added if g i is achieved before g j .
¯ G is randomly shufﬂed at the start of an episode such that every partial order is satisﬁed.
If every goal is achieved, then the total order is known though it might not be optimal.
However, due to exploration in RL, the order of goals achieved might not be the same as ¯ G .
Thus, a more optimal ¯ G could still be found.
Example 2.
In
SR , the goals are represented by the state predicates goal object with ( obj, person ) and goal object at ( obj, wp ) .
Suppose that the goals are to deliver item O 1 to location WP 5 and item O 2 to person P 1 .
The former is denoted as g 1 = goal object at ( O 1 , WP 5) and the latter as g 2 = goal object with ( O 2 , P 1) .
The goal g 1 involves the objects O 1 with type obj and WP 5 with type wp , and the corresponding contextual grounding is σ goal =
g 1 = { (cid:63)obj/O 1 , (cid:63)wp/WP 5 } .
Likewise, σ goal = g 2 = { (cid:63)obj/O 2 , (cid:63)person/P 1 } .
Location Context.
Mobile robots move in an environment and can only interact with objects in their vicinity.
Following this observation, location context substitutes the free variable with the current location of the robot: σ loc = { (cid:63)wp/WP } if the robot is at location WP of type wp .
Location context can be used with the goal context if there is no conﬂict (i.e., they substitute free variables of different types)—this queries if the robot is at the same location as the goal of interest. If there is a conﬂict between goal context and location context, then either one takes precedence.
We let location context take precedence in our experiments.
Example 3.
In SR , the state predicate robot at ( wp ) represents the location of a robot.
If the robot is at WP 4 , then the contextual grounding is σ loc =
{ (cid:63)wp/WP 4 } .
Following Example 2, we can apply σ goal =
g 2 and σ loc together without any conﬂict to give the contextual grounding σ loc,goal = g 2 = { (cid:63)wp/WP 4 , (cid:63)obj/O 2 , (cid:63)person/P 1 } .
On the other hand, σ goal = g 1 and σ loc are conﬂicting as they ground (cid:63)wp to WP 5 and to WP 4 , respectively.
They can still be applied together by letting either one takes precedence (i.e., ignore the grounding of the other one).
If location context takes precedence, then σ loc,goal = g 1 = { (cid:63)obj/O 1 , (cid:63)wp/WP 4 } .
4.4 Granularity of First-Order Approximations
The lifted state-action space (cid:98) S × (cid:98) A is often much smaller than the ground state-action space since a lifted predicate p ( x, y ) can be ground in | x
|| y | ways.
Since (cid:98) S , (cid:98) P , and (cid:98) A are independent of O , the lifted state-action space does not increase with the scale of the planning problem which makes ﬁrst-order approximations scalable to large scale problems (see Example 1).
However, the downside is a coarser granularity in approximation.
The maximum number of features is (cid:80) a ∈
A | Φ ( a ) | = 2 2 | P | ×| A | in a ground approximation, and (cid:80) ˆ
a ∈ (cid:98) A | Φ ˆ a | = 2 2 | (cid:98) P | ×| (
cid:98) A | in a ﬁrst-order approximation.
Since | (cid:98) P | ≤ | P | and | (cid:98)
A | ≤ | A | (they are equal if there is one object for each type), a ﬁrst-order approximation uses fewer features to partition the state space.
Therefore, the size of the partitions (or granularity) must be larger than a ground approximation.
Furthermore, free variables in a ﬁrstorder feature increase its coverage because there only needs to exist a substitution among several possible substitutions to make the feature true.
This deteriorates performance if there are plateaus in the induced policy.
Plateaus are regions of the state space where there are multiple non-optimal actions with the maximal Q-value.
The coarse granularity limits the type of problems a ﬁrst-order approximation can be applied in.
Because ﬁrst-order approximation does not consider every ground state predicates, it is suited for planning in factored MDPs with independent goals.
[Table]  Experiments and Results
We now demonstrate empirically that our work (1) learns a generalised and goal-directed policy, (2) enables transfer learning from small scale problems ( P small ) to large scale problems ( P large ) and between different simulation environments, and (3) reduces memory and computational costs.
Results are averaged over 10 independent runs where a different, randomised problem is used.
The problems are randomised in the locations of items and in the tasks given.
We assume that the preconditions of actions are known (i.e., an illegal action will never be executed).
Experimental Setup.
In P small ( P large ), O contains three (six) items, one person (three people), and ﬁve (ten) locations; the probability of people needing assistance is 0.5 (0.5, 0.3, and 0.0 for three people). The size of the state-action spaces are 2 56 × 65 for P small and 2 216 × 264 for P large .
Each person who requires assistance gives two tasks when talked to.
Executing an action provides a reward of − 1 and completing a task (or goal) provides a reward of 20 .
We tested our work in two simulation environments: RDDLSim [27] and ROS (Robot Operating System) [28] with the ROSPlan [29, 30] package.
Both simulators return an observation for executing an action in a state (line 4 of Algorithm 1).
We augment the observation from RDDLSim with the execution duration ∆ which is the mean of the observed ∆ in ROS injected with normally distributed noise (standard deviation is 20% of the mean).
We built a Gazebo environment (see Figure 1) for a more realistic simulation relative to RDDLSim .
Firstly, in RDDLSim , the robot loses localisation when moving with a probability of 0.1; in ROS , it is deemed to have lost localisation when the covariance of the pose estimation from adaptive Monte-Carlo localisation exceeds a user-deﬁned threshold.
Secondly, in RDDLSim , the robot always succeeds in ﬁnding a person and ends up in the same location as the person.
In ROS , the robot follows an exploration path to ﬁnd a person.
It might ﬁnd the intended person and/or other people it encounters along the way, and might not be in the same location as the person since it could detect people from a distance.
Lastly, in RDDLSim , picking up and putting down an item always succeeds while these actions might fail in ROS .
Each experiment in RDDLSim is conducted on a single core AMD Opteron 6376 and with 8 GB of RAM.
Ablation Study.
We conducted an ablation study for contextual grounding in ﬁrst-order approximation and benchmarked it against ground approximation.
RDDLSim was used.
The results are shown in Figure 2.
We measure performance by the total undiscounted reward received in each episode and by the total duration of plan execution (mission time).
Executing an action incurs a reward of −which could result in a learned policy which minimises the number of actions executed; this seems to sufﬁce in reducing the mission time as it was observed that the mission time reduces asymptotically while the reward increases.
G1 and Loc.
+ G1 (combination of location context and G1 ) generally outperforms GA and Loc.
+ GA .
This is because the goals in SR are independent of each other and considering one goal at a time reduces σ Φ ˆ a and the coarseness of granularity more than GA does.
In P small , using no context or only location context has the worst performances.
Due to our modelling choice to include wp in arguments of actions (cid:0) e.g., pick up ( obj, wp ) and put down ( obj, wp ) (cid:1) , features with (cid:63)wp are unnecessary to approximate the Q-values of some actions—we opt against using domain knowledge to prune these features.
Nevertheless, location context is crucial in reducing the memory and computational cost; experiments for P large without location context are computationally expensive and are omitted.
The computation time for P large is further reduced when goal context is also used.
The computation time for P small is comparable for all runs and is thus omitted.
Although ground approximation has the best performance in P small , it has the worst performance and the highest computational cost in P large . The former is due to a lack of generalisation and the latter is due to the large set of features.
For P small , ground approximation has 6894 ± 2 .
7 (one standard deviation) features while the various conﬁgurations of ﬁrst-order approximation haveto 316 features. In contrast, for P large , ground approximation has 94 . 1 × 10 3 ± 8 . 1 × 10 3 features while the various conﬁgurations of ﬁrst-order approximation have 382 to 417 features.
Evidently, ground approximation has poorer scalability than ﬁrst-order approximation.
Transfer Learning.
In Figure 2, we show the results of transfer learning where (cid:101) Q trained in P small (cid:0) (cid:101) Q P small (cid:1) is used, and continues to be updated in P large .
The type of contextual grounding used in P small must also be used in P large for transfer learning to work.
We consider only Loc.
+ G or Loc.
+ GA as these are computationally tractable.
The former outperforms the latter due to the aforementioned reason.
The boost in performance due to transfer learning is evident in episode 1 .
The asymptotic performance without transfer learning approaches that of transfer learning as expected.
However, for Loc.
+ GA , transfer learning performed asymptotically poorer than without transfer learning.
This is because GA considers multiple goals for contextual grounding.
Thus, it generalises poorly from P small , which has two goals, to P large , which has four goals.
Lastly, we evaluate the generalisation from RDDLSim simulations to ROS simulations.
We used ten randomised problems for P small and P large which are different from the ones used in RDDLSim .
A greedy policy induced by (cid:101) Q P small using Loc.
+ G1 can directly solve P small but not P large .
Therefore, we trained (cid:101) Q P small further in 500 episodes of P large using RDDLSim with a modiﬁed transition function (i.e., the robot’s location does not change when ﬁnding a person) to solve P large .
Without further training, the policy could fail to instruct the robot to move to a person after ﬁnding that person.
The results are shown in Table 1 alongside the results from RDDLSim (i.e., Figure 2) for ease of comparison.
For P small , the performance in RDDLSim is better with smaller variances which is expected since the simulations do not consider real-world dynamics and are more optimistic.
The results for P large are comparable for both simulation environments.
Thus, the ﬁrst-order approximations are successful at generalising over both simulation environments.
Conclusion
In this paper, we proposed a model-free, online RRL algorithm which learns an approximation of the Q-function with ﬁrst-order features that is able to generalise across different problems of the same domain.
Free variables in the ﬁrst-order features caused ambiguity in how they should be substituted.
The choice of a substitution inﬂuences the learning and is critical to performance.
We resolve this problem by utilising contextual knowledge to substitute free variables.
Empirical results showed that our method is scalable to large scale problems by keeping memory usage and computational cost tractable, and can generalise across problems of different scales and across different simulation environments. In future work, we plan to investigate other forms of contextual knowledge. Acknowledgments
This work was partially funded by the EPSRC ORCA Hub (http://orcahub.org/) under grant number EP/R026173/1. A Appendix A.1 Relational Reinforcement Learning We elaborate on our relational reinforcement learning method (an abridged version is outlined in
Algorithm 1) which is shown in Algorithm 2.
The inputs to the algorithm are a RMDP representing a planning problem, a set of features Φ and weights w 0 which approximate the Q-function (cid:101) Q , candidate features Φ c with their relevance η , the discovery threshold ξ for iFDD+ , learning rate α , contextual knowledge CK (if any), and a maximisation metric M .
Φ , w 0 , Φ c , and η are learned in another planning problem and transferred to the current problem.
If transfer learning is not used, then they are the empty set ∅ and need to be initialised (lines 1 to 5).
The initial set of ﬁrst-order features Φ is the concatenation of ﬁrst-order features for each lifted action, Φ ˆ a .
Φ ˆ a is initialised with Algorithm 4 (line 4) which is described later.
The weights are initialised to a vector of zeroes (line 5).
In a linear function approximation, the Q-value of a state-action pair ( s, a ) is the dot product of w and Φ ( s, a ) (line 6).
The latter is returned by Algorithm 3 which is described later.
Lines 7 to 23 show the planning cycle for H time steps.
To select an action using a policy π (cid:101) Q generated by (cid:101) Q (line 8), the Q-values of every action must be evaluated.
This implies that Φ ( s t , a ) must be evaluated for every a ∈ A .
Thus, Algorithm 3 is called | A | times.
The action returned by π (cid:101) Q is executed (line 10) which returns an observation ( s t +1 , r t , ∆ t ) where s t +1 is the successor state, r t is the reward, and ∆ t is the duration of the execution.
The TD error is computed (line 12) and the weights are updated (line 13) following the observation.
The vector of real numbers, Φ ( s t , a t ) (line 11), has a value of 1 if the corresponding feature is active.
The set of active features is extracted (line 14).
The set of active candidate features is the set of conjunction of every pair of active features in Φ (line 15).
For each active candidate feature, its relevance is computed (line 17).
Candidate features with relevances greater than ξ are added to Φ with initial weights equal to the sum of the weights of their parent features (lines 18 to 20).
New candidate features can be constructed from these newly added features in the next time step.
The set of candidate features accumulated thus far, Φ c , and its relevances, η , are updated accordingly (lines 21 and 23).
The algorithm terminates after H time steps or if the terminal state is reached (line 7).
It returns w , Φ , Φ c , and η (line 24) which are used as inputs for the next planning problem (i.e., transfer learning).
Algorithm 3 evaluates the binary value of each feature φ f ∈ Φ and returns a vector of real numbers.
The set of possible substitutions σ Φ consists of every combination of objects for the free variables (line 2).
We apply contextual grounding (line 3) by considering the state s (e.g., to determine the goal(s) and the location of the agent).
Then, a selection method selects a subset of substitutions from σ Φ ∩ σ CK (line 4) such that the metric M is maximised.
Φ σ represents the grounding of Φ with σ .
Lastly, an element-wise logical disjunction operator is applied to every grounding of Φ with σ M (line 5).
A.2 Initialisation of First-Order Features
We describe the initialisation of a set of ﬁrst-order features, Φ ˆ a , for each lifted action ˆ a as shown in Algorithm 4.
The inputs are the set of state ﬂuents P and the set of ground actions for ˆ a , A ˆ a .
For each ground action a ∈
A ˆ a , a set of ground features Φ a is initialised ( initialise ground features in line 4).
In this work, we used the set of every state ﬂuent and their negation, as well as the non-ﬂuents.
Each feature φ f ∈
Φ a is lifted in accordance with the substitution of a , σ a ( lift in line 5).
A feature might be partially lifted if it contains objects that are not in the arguments of a .
Φ ˆ a is the union of these partially or fully lifted features for each a ∈
A ˆ a (line 5).
Remaining objects in Φ ˆ a are substituted with free variables to yield a set of fully lifted, or ﬁrst-order, features ( quantify in line 6).
A.3 Domain Model
RDDL [27] is a planning language for describing planning problems, and is used in recent International Probabilistic Planning Competitions (IPPCs) [31, 32].
Semantically, RDDL describes dynamic Bayesian Networks extended with an inﬂuence diagram.
A RDDL domain is described by object types, non-ﬂuents, ﬂuents, conditional probability functions (CPFs), and a reward func-
Input: RMDP ( O , P , S , A , T , R , s 0 , H, γ ) ,
Features Φ ,
Weights w 0 , Candidate features Φ c with relevance η ,
Discovery threshold ξ ,
Learning rate α ,
Contextual knowledge CK ,
Maximisation metric M 1 if then (cid:98)
A
A for ˆ a
(cid:98) A do Φ ← Φ initialise
( P , a ) 5 if w 0 =
then w 0 ←6 (cid:101) Q ( s, a ) :
= w T Φ ( s, a
Φ ( s a ) ← evaluate features ( O , Φ , CK, M, s, a ) 7 for t = 0 to H − s t is not terminal state do a t ← π (cid:101) Q ( s t ) ˆ a t ← lift ( a t ) s t +1 , r t , ∆ t ← Execute a t in simulation environment or real-world Φ ( s t , a t ) ← evaluate features ( O , Φ , CK, M, s t , a t ) δ t =
r t + γ max a ∈ A (cid:101) Q ( s t +1 , a ) − (cid:101) Q ( s t , a t )
{+Update w : w it +
+}
=
w it + αδ t
φ
i ( s,a ) || Φ ( s,a ) || 1 ∀ w it ∈ w Φ active which active in s Φ cactive =
P ( active ) for φ
i
Φ cactive do
Relevance η t ( φ i ) = (cid:12)(cid:12) (cid:80) tj =0 ,φi ( sj,aj )=1 δ j (cid:12)(cid:12) (cid:114)(cid:80) tj =0 ,φi ( sj,aj )=1 if η t ( φ i ) > ξ then
Append φ i to Φ ˆ a ⊂ Φ
Append w i = w j + w k to the sub-vector of w for ˆ a t where φ i = φ j ∧ φ k
Remove φ i from Φ c if φ i ∈
Φ c and remove η t ( φ i ) from η
if η t ( φ i ) ∈
η else
Add φ i to Φ c if φ i (cid:54)∈ Φ c and add/update η t ( φ i ) in η 24 return w , Φ , Φ c , η
Function evaluate features( O , Φ , CK, M, s, a ) :
Input: Set of objects O ,
Features Φ ,
Contextual knowledge CK ,
Maximisation metric M ,
State s ,
[Table]
a
[Table]
from
[Table]
Φ
[Table]
tion.
State ﬂuents (non-ﬂuents) are state variables that change (do not change) with time.
A RDDL problem is speciﬁed by objects, initial state, and values of non-ﬂuents.
A domain can have different problems by randomising the initial state and non-ﬂuents.
We have included the RDDL domain and problems ﬁles for our Service Robot ( SR ) planning problem as supplementary material.
We brieﬂy describe the state ﬂuents, non-ﬂuents, and actions for SR .
We omitted the argument robot for brevity as SR is a single-agent problem but included them here for consistency with the RDDL domain and problem ﬁles.
Function initialise features( P , A ˆ a ) :
Input: Set of state predicates P ,
Set of ground actions A ˆ a Φ ˆ a = ∅ for a ∈
A ˆ a do Φ a ← initialise ground features ( P , a ) Φ ˆ a ← Φ ˆ a ∪ lift (cid:0) Φ a , σ a (cid:1) return quantify (cid:0)
Φ ˆ a (cid:1)
A ﬂuent or non-ﬂuent is represented by its name and typed variables in its arguments.
We use the type robot to represent a robot, wp to represent a location, obj to represent an item, and person to represent a person.
The state ﬂuents which describe the states and goals are:
• robot at ( robot, wp ) : robot is at location wp , • localised ( robot ) : robot is localised, • emptyhand ( robot ) : robot is not holding any items, • holding ( robot, obj ) : robot is holding item obj , • object at ( obj, wp ) : item obj is at location wp , • object with ( obj, person ) : item obj is with person , • goal object at ( obj, wp ) : a goal is for item obj to be at location wp , • goal object with ( obj, person ) : a goal is for item obj to be with person , • person at ( person, wp ) : person is at location wp , • need assistance ( person ) : person needs assistance, • needed assistance ( person ) : person needed assistance (a person only needs assistance once per problem), and • reward received ( obj ) : the goal involving obj has been achieved (each obj can be involved in at most one goal).
We use non-ﬂuents to represent the location of each person and the tasks (goals) they would give.
These non-ﬂuents are not made known to Algorithm 2.
The non-ﬂuents are:
• PROB NEED ASSISTANCE ( person ) : the probability of person needing assistance, • PERSON GOAL OBJECT AT ( person, obj, wp ) : person will instruct the robot to place item obj at location wp , • PERSON GOAL OBJECT WITH ( person 1 , obj, person 2 ) : person 1 will instruct the robot to bring item obj to person 2 where person 1 and person 2 could refer to the same person, and • PERSON IS AT ( person, wp ) : person is at location wp (this is different from person at ( person, wp ) which represents the knowedge of the robot).
The actions which the robot could execute are:
• move ( robot, waypoint 1 , waypoint 2) : robot moves from waypoint 1 to waypoint 2 , • localise ( robot ) : robot localises by performing simultaneous localisation and mapping (SLAM), • find person ( robot, person ) : robot explores the room to ﬁnd person (this action terminates at the end of the exploration path or when person is found), • talk to person ( robot, person ) : robot talks to person (cid:0) if need assistance ( person ) is true, then two goals from person will be made known and are represented by goal object at ( obj, wp ) and/or goal object with ( obj, person ) (cid:1) , • pick up ( robot, obj, wp ) : robot picks up item obj from location wp ,
• put down ( robot, obj, wp ) : robot puts down item obj at location wp , • take ( robot, obj, person ) : robot takes item obj from person , and • give ( robot, obj, person ) : robot gives item obj to person .
A.4 Experiment Details
In our experiments, we used an (cid:15) - greedy policy with a linearly decaying (cid:15) over episodes.
The parameters used are (cid:15) initial = 1 for experiments without transfer learning and 0 .
otherwise, α = 0 .
3 (learning rate), γ = 0 .
9 , λ = 0 .
7 (decay rate for eligibility traces), and ξ = 3 .
H is 20 ( 40 ) for P small ( P large ).
We utilise two simulators in our experiments: RDDLSim [27] and ROS [28].
RDDLSim returns a successor state and reward based on the CPFs and reward function deﬁned in the RDDL domain.
It does not simulate a robot operating in an environment.
For a more realistic simulation, we use ROS which considers aspects such as sensors (for localisation, navigation and detecting people), motion planning (for navigation, manipulation planning (for grasping), etc.
For manipulation planning, we observed that the robot fails to grasp the item frequently.
Since our work does not deal with manipulation planning, we simulated the success of grasping.
If the robot fails to grasp an item, we would still consider it a success with a probability of 0 . 8 and update the successor state accordingly (e.g., holding ( robot, obj ) is true even if the robot fails to pick up an item obj in
ROS ).
Regardless, we observed that our learned policy will repeatedly attempt to pick up an item if it failed to in previous attempts.
A.5 Additional Results for the Service Robot Domain
In Section 5, we present results for small scale (one person , three obj , and ﬁve wp ) and large scale (three person , six obj , and 10 wp ) problems.
Here, we denote the former as SR1 and the latter as SR2 .
In SR1 , person p 1 needs assistance with a probability of 0.5.
In SR2 , person p 1 needs assistance with a probability of 0.5, person p 2 needs assistance with a probability of 0.3, and person p 3 does not require assistance.
We now consider a variant of SR2 where p 3 also needs assistance with a probability of 0.3.
We denote this variant as SR3 .
For SR1 , SR2 , and SR3 , the problems are randomised in the initial locations of objects and the types of goals:
• PERSON GOAL OBJECT
AT ( p 1 , obj, wp ) : a goal from p 1 where obj and wp are randomised object instances, • PERSON GOAL OBJECT WITH ( p 1 , obj, p 1) : a goal from p 1 where obj is a randomised object instance, • PERSON GOAL OBJECT
AT ( p 2 , obj, wp ) : a goal from p 2 where obj and wp are randomised object instances, • PERSON GOAL OBJECT
AT ( p 2 , obj, wp ) : a goal from p 2 where obj and wp are randomised object instances,
• PERSON GOAL OBJECT WITH ( p 3 , obj, p 1) : a goal from p 3 where obj is a randomised object instance, and • PERSON GOAL OBJECT WITH ( p 3 , obj, p 2) : a goal from p 3 where obj is a randomised object instance.
The goals are randomised such that no obj is involved in more than one goal.
The goals from pinvolve another person (i.e., p 3 requires the robot to bring obj to the others).
The results for SR are shown in Figure 3.
The ﬁndings are similar to those discussed in Section 5 (see Figure 2): (1) the use of contextual grounding reduces computation time and improves performance, (2) transfer learning from a small scale problem (i.e., SR1 ) to a large scale problem (i.e., SR3 ) gives an initial performance boost, and (3) ground approximation does not scale well to large scale problems.
A.6 Ablation Study for IPPC Domains
The IPPCs use several planning domains as benchmarks to evaluate the performance of competing planners.
We conducted an ablation study on three benchmark domains: Recon ( RC ) [31],
Academic Advising ( AA ) [33], and Triangle Tireworld ( TT ) [34].
These domains have problems which are numbered from 1 to 10 (a larger number represents a problem with a larger scale).
We use Domain# to denote a problem numbered # for Domain .
We conducted experiments for both small and large scale problems for each domain.
A Q-function is learned in the small scale problems and transferred to the large scale problems.
The results are shown in Figure 4.
Since these domains do not involve durative actions, there are no results for mission time.
The simulator used is
RDDLSim .
In RC , an agent moves in a grid environment where there is a base, hazard, and objects.
The goal is to use the tools to get readings on objects; tools can be damaged if the agent is at or adjacent to a hazard and this reduces the probability of getting a good reading.
The agent can repair a tool at the base.
We replaced the actions up , down , left , and right with move ( wp ) .
This does not simplify the problem but allows separate weights for each ground action of move .
We used RC3 and RC6 , where the size of the state-action spaces are 2 42 × 28 and 2 55 × 38 , respectively.
In AA , a student has to pass some required courses.
The passing rate of a course course 1 depends on the number of prerequisites course 2 the student has passed.
A reward is given at each time step for taking a course ( − 1 ) or retaking a failed course ( − 3 ), and if any required course has not been passed ( − 5 ).
Passing a required course gives a reward of 5 .
We used AA3 and AA5 , where the size of the state-action spaces are 2 30 × 16 and 2 40 × 21 , respectively.
In TT , a vehicle moves in a grid environment to reach a goal location.
There is a probability of 0 .of getting a ﬂat tire when moving.
The tire needs to be replaced with a spare tire; if there isn’t one, a deadend is reached.
The vehicle can load a spare tire if it doesn’t have one and there is a spare at its current location.
We used TT3 and TT6 , where the size of the state-action spaces are 2 33 ×and 2 59 × 814 , respectively.
Ablation Study.
We investigate the utility of contextual grounding by benchmarking the performance against two baselines: ground approximation and ﬁrst-order approximation without contextual grounding.
In RC6 , ﬁrst-order approximation outperforms ground approximation.
The experiments for ﬁrst-order approximation without contextual grounding could not be completed due to high computational costs.
In AA5 , ﬁrst-order approximation performs poorly.
This is because the goals could be dependent on other goals—the success of passing a course depends on the number of prerequisites passed.
This cannot be represented by binary features.
This issue is mitigated in a ground approximation due to the inclusion of every ground state ﬂuent as features which serves as implicit counting.
In TT6 , the use of location context or goal context (they cannot be used together in this domain) improved the performance of ﬁrst-order approximation which performed comparably to ground approximation.
Next, we consider the computational time.
In RC6 , location context and the combination of location and GA context have the highest computation time.
G1 considers only one goal at a time while
GA considers every unachieved goal.
The former has a signiﬁcantly faster computation time.
In
AA5 , the highest computation time is incurred when no context is used.
Since contextual grounding reduces the set of possible substitutions for Φ , the computation time decreases sharply.
The same observation is seen in TT6 .
Knowledge Transfer.
We used a greedy policy generated by the Q-functions learned in the small scale problems (i.e., RC3 , AA3 , and TT3 ) to solve the large scale problems (i.e., RC6 , AA5 , and
TT6 ).
The Q-functions are kept unchanged while solving the large scale problems (i.e., knowledge transfer where no learning takes place).
This is different from the setup in SR where an (cid:15) -greedy policy is used and the Q-function continues to be updated (i.e., transfer learning).
Figure 4 shows that in all problems, knowledge transfer performed signiﬁcantly better than those which learn the Qfunction online from scratch (i.e., online RL).
Since the Q-function is not updated, the performance of knowledge transfer does not improve over episodes.
Thus, in RC6 and AA5 , they are outperformed by online RL in later episodes.
The computation time for knowledge transfer is lower than online
RL as expected since no learning is done in the former.