Generalised Task Planning with First-Order Function Approximation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Real world robotics often operates in uncertain and dynamic environments.
Therefore, generalisation over different scenarios is of practical interest.
In the absence of a model, value-based reinforcement learning can be used to learna goal-directed policy.
Typically, the interaction between robots and the objectsin the environment exhibit a ﬁrst-order structure.
We introduce ﬁrst-order, or relational, features to represent an approximation of the Q-function so that it caninduce a generalised policy.
Empirical results for a service robot domain showthat our online relational reinforcement learning method is scalable to large-scaleproblems and enables transfer learning between different problems and simulationenvironments with dissimilar transition dynamics.
Introduction
Adaptation to different scenarios is essential for real world robotic applications.
As a motivatingexample, consider a service robot interacting with people and objects as shown in Figure 1.
Therobot is required to attend to a person who requires assistance and complete the tasks given bythe person.
The uncertainty in who needs assistance, what these task could be, and the numberof people and objects in the environment necessitates generalisation of learned knowledge frompast observations in order to complete new but similar tasks in a different environment.
If themodel of the interaction between the robot and its environment is unknown due to its complexity,then value-based reinforcement learning (RL) methods can be used to solve the planning problems.
The amount of observations required to achieve near-optimal behaviours, or sample complexity, isusually prohibitive, especially in large-scale problems.
This restricts the practicality of RL methodsin applications where data collection is expensive.
A solution is to utilise transfer learning whichcan reduce sample complexity by leveraging the knowledge learned in small-scale problems wherelearning can be more efﬁcient, or in simulated environments where data collection is cheap and safe.
Interactions between robots and objects in its environment often exhibit a ﬁrst-order structure. Wemodel planning problems with a ﬁrst-order structure with Relational Markov Decision Processes(RMDPs). Relational reinforcement learning (RRL) learns in a ﬁrst-order representation rather thana propositional representation and can achieve knowledge transfer without the need of a mappingbetween problems [1].
We propose a model-free, value-based RRL method where the action-valuefunction, or Q-function, is approximated by projecting the state space into a lower dimensionalspace using a set of features.
We utilise an existing online feature discovery algorithm, iFDD+ [2], toincrementally add conjunctive state predicates as features to reduce the approximation error.
Sincestate predicates are ground over objects, problems with different objects or numbers of objects havedifferent sets of state predicates, or feature spaces, and generalisation is not possible.
Instead, we uselifted state predicates, or ﬁrst-order features, to approximate the Q-values of lifted actions rather thanground actions.
By learning in a abstract state-action space which is common across all problems ofthe same domain ( related problems ), knowledge transfer is possible.
Our contribution is threefold. First, we propose a method to generate a ﬁrst-order feature space automatically given a RMDP without the use of a model, expert knowledge, or training data. Second, we implement a online, model-free RRL method which learns a ﬁrst-order linear function approximation of the Q-function given an initial set of ﬁrst-order features.
The approximated Q-functioninduces a generalised policy which allows transfer learning between related problems independentof the objects, number of objects, initial states, and goal states.
Third, we introduce the concept ofcontextual knowledge to reduce the granularity in the ﬁrst-order function approximation, improveplan optimality, and reduce computational cost.
We evaluate our method empirically on randomisedproblems of a service robot domain and in different simulation environments.
Related Work
Exploitation of the ﬁrst-order structure of planning problems could reduce the sample complexity of RL methods.
There are broadly two types of methods: model-based and model-free.
Model-basedmethods such as [3, 4, 5, 6] learn relational models and employ planning techniques.
Our work is amodel-free RRL method.
Therefore, we focus on reviewing model-free learning methods.
Mausamand Weld [7] and D ˇ zeroski et al.
[8] partition the state space into ﬁner regions, each of which hasa real-value representing the Q-value, using relational regression trees.
Wu and Givan [9] performsupervised learning of relational features, represented as decision trees, by adding features whichcorrelate well to the Bellman error of value functions.
A decision tree is sensitive to the order ofnode splitting and is ill-suited in online RL where later observations yield new information whichmight necessitate the reconstruction of the trees.
Ramon et al.
[10] propose a tree restructuringoperation but require statistics to be stored for every node.
Croonenborghs et al.
[11] learn relationaloptions using the relational regression trees from [8] which classify if a ground action is an actionfor the policy of a relational option.
We use conjunctive features similar to SVRRL [12] which avoidsthe issue faced by decision trees.
SVRRL represents a value function with a relational naive Bayes netand learns both the values and structure of the network.
It utilises a distance metric to generalise overhandcrafted non-binary relational features.
Likewise, RIB [13], which uses instance-based learningwhere selected observed examples are stored, requires a domain-speciﬁc distance metric to compute
Q-values of unseen state-action pairs.
Our work uses state predicates as features and does not requireany distance metric to be deﬁned.
In these aspects, [14] and [15] are most similar to our work.
Theformer samples features from a set of features with the use of prior training data while we discoverfeatures online.
The latter restricts features to have at most one free variable and approximates thevalue function while our features can have any number of free variables and we approximate the Q-function.
Other works such as [8, 10, 11, 13] do not consider free variables. Instead, only objectsin the arguments of the goal predicate and action are considered which restricts the applicability oftheir methods to problems with simple goals.
Some works require extensive domain knowledge to deﬁne speciﬁc representations to facilitate efﬁcient learning.
Morales [16] deﬁnes abstract actions (r-actions) and abstract states (r-states) ina relational representation, then uses a modiﬁed Q-learning to learn policies based on the abstractstate-action space.
Konidaris and Barto [17] learns a shaping reward in the action-space, representedby sensations which are common and hold the same semantics for all problems, to accelerate learning in another problem.
In a later work [18], options in the agent-space are learned and transferred.
Input: RMDP ( O , P , S , A , T , R , s 0 , H, γ ) , Discovery threshold ξ 1 Initialise: Φ ← initialise features ( P , A ); w ← 0 ; t ← 0 ; (cid:101) Q := w T Φ 2 for t = 1 to H and s t is not terminal state do a t ← π (cid:101) Q ( s t ) , ˆ a t ← lift ( a t ) s t +1 , r t , ∆ t ← Execute action a t in simulation environment or real-world
Compute temporal difference (TD) error δ t ← r t + γ max a ∈ A (cid:101) Q ( s t +1 , a ) − (cid:101) Q
[Table]
s t , a t )
; Update w
[Table]
Van Otterlo [19] creates an abstract MDP of the underlying RMDP with the use of backgroundknowledge, then solves the abstract MDP with modiﬁed Q-learning or prioritized sweeping.
Sharmaet al.
[20] perform case-based learning and quantify the state similarity by the Euclidean distancebetween a pair of states which are represented by real-valued state variables.
These state variablesare handcoded and can describe any state in all problems for their domain of interest.
The Q-valuesof an action is a weighted sum of the Q-values of matching cases, weighted by how similar thesecases are to the current state.
Our work uses trivial domain knowledge for contextual groundingwhich we posit are known in most, if not all, planning problems anyway.
Preliminaries
Relational Markov Decision Process (RMDP).
Factored Markov Decision Processes (MDPs)
[21]represent a state s with a set of state predicates P where s = { p i } | P | i =1 and p i ∈ P .
We usesymbols with boldface to indicate sets.
If the transition of p i depends only on a small number ofstate predicates ¯ P ⊂ P , then the transition function T can be represented compactly; for a predicatep
i , T ( p i ' | P , a ) =
T ( p i ' | ¯ P , a ) where p i ' is the predicate at the next time step and a is an action.
A Relational Markov Decision Process (RMDP) is a ﬁrst-order representation of a factored MDPgiven by the tuple ( O , P , S , A , T , R , s 0 , H, γ ) .
O is a set of objects, each associated with a type,
P is deﬁned over O , S is a set of all possible state speciﬁcations over O and P , A is the set of allpossible instantiated actions, T : S ×
A × S → [0 , 1] , R : S ×
A → R is the reward function, s 0is the initial state, H is the planning horizon, and γ is the discount factor.
Each predicate is appliedover a type-consistent tuple of objects.
Different planning problems of a domain can be constructedwhere their RMDPs have different O , initial states, and goal states.
A generalised policy is a policyπ : s → a which directly solves any problem of a domain.
Linear Function Approximation.
The Q-function estimates the expected return, or Q-value Q ( s, a ) , of executing an action a in state s and following a policy thereafter.
The Q-function canbe represented with some form of parametric function approximation such as a linear combinationof features: (cid:101) Q
= (cid:80) k
i = 1 w i φ i where Φ = { φ 1 , . .
. , φ k } is a set of features and w = { w 1 , . .
, w k }is a set of scalar weights.
A feature φ i maps ( s, a ) to a scalar.
The learning objective is to ﬁnd theset of features and weights such that (cid:101) Q closely approximates the optimal Q-function.
A policy πinduced by the optimal Q-function maximises the total discounted reward (cid:80) Ht
=
γ t r t where r t is thereward observed at time step t .
A feature partitions the state space into regions such that states ina region are considered the same for the purposes of approximating the Q-values.
The coverage ofa feature is the region of the state space for which the feature evaluates to non-zero [22].
Featureswith low coverage give ﬁne granularity in approximation and thus have better accuracy than featureswith high coverage.
This impacts the soundness of policies.
Conversely, features with high coverageoffer better generalisation as learning is done over a smaller number of partitions of the state space.
Online Learning with First-Order Linear Function Approximation
Our objective is to learn a generalised policy which can be used to directly solve every problem ofthe same domain.
We present our online RRL method in Algorithm 1 which learns a linear functionapproximation of the Q-function, (cid:101) Q , where the features Φ are ﬁrst-order (initialised in line 1). (cid:101) Q is deﬁned over the abstract state space (represented by the ﬁrst-order features) and lifted action spacewhich are the same in all related problems.
This allows transfer learning where (cid:101) Q is learned in aproblem and transferred directly to solve another problem. Thus, the policy π (cid:101) Q induces by (cid:101) Q is ageneralised policy.
At each time step t (line 2), the robot executes an action a t given by π (cid:101) Q (lines3 and 4) and observes the successor state s t +1 , reward r t , and duration of execution ∆ t .
The TDerror is computed and the weights w are updated accordingly (line 5) with a TD learning method.
We used Double Q-learning [23] with replacing eligibility traces for each feature [24].
Using iFDD+[2] for online feature discovery, new features, which are the conjunction of two existing features inΦ , are added to Φ if their relevances η t exceed the user-deﬁned threshold ξ (see Section 4.2).
Thisreduces the approximation errors of (cid:101) Q by introducing a ﬁner granularity in (cid:101) Q .
In the remainder ofthis section, we elaborate on Algorithm 1.
4.1 Generalisation with First-Order Features
A feature φ i is a set of conjunctive state predicates or a state predicate and evaluates to 1 if itis satisﬁed in a state, otherwise it evaluates to 0 .
We denote a function approximation which usesground state predicates for features as a ground approximation .
Let (cid:101) Q 1 be a ground approximationfor a RMDP with state-action space S 1 ×
A 1 and φ 1 i be one of its features.
Evaluating the value ofφ 1 i = p 1 m ∧ . . .
∧ p 1 n ∈ S 1 is straightforward.
However, in another RMDP (i.e., a different planningproblem) with state-action space S 2 × A 2 , φ 1 i cannot be evaluated if any of its constituent statepredicates p 1 m , ..., p 1 n is not in S 2 .
Extending this further, (cid:101) Q 1 ( s, a ) does not exist if s ∈ { S 2 − S 1 }or a ∈ { A 2 − A 1 } .
In other words, S 2 × A 2 does not lie within the feature space of (cid:101) Q 1 .
Thus,(cid:101) Q 1 cannot induce a policy for the second RMDP unless S 1 = S 2 and A {+
+} =
A 2 .
However, the two RMDPs are trivially identical with possibly different initial states and goal states.
We are interestedin transfer learning between related problems with different state-action spaces.
This motivates ouruse of a ﬁrst-order approximation which is independent of S , A , and O .
In a ground approximation, the features for each ground action a , Φ a , are initialised as the set ofevery state predicate and their negation; Φ :=
[ Φ a 1 , ..., Φ a m ] where m is the number of groundactions. We use ˆ to denote a lifted predicate or a set of lifted predicates and lowercase (uppercase)letters to represent variables (objects).
In a ﬁrst-order approximation, we instantiate a set of ﬁrstorder features for each lifted action ˆ a instead of every ground action. Otherwise, (cid:101) Q will not beindependent of O because ground actions are deﬁned over O .
ˆ a can be ground with the variablebinding σ a = { x/X, y/Y } to a ground action a ( X, Y ) .
For every ˆ a ∈ (cid:98) A , we initialise Φ ˆ a (line 1of Algorithm 1) with the following steps:
We denote the set of ground actions for ˆ a as A ˆ a .
The initial set of ground features for every ground action a ∈
A ˆ a are lifted in accordance with σ a where objects in the arguments ofa are substituted with their corresponding bound variables .2.
Φ ˆ a is the union of the features from step 1.
Substitute remaining objects with free variables .
We use (cid:63)obj to denote a free variable of type obj .
This procedure will generate the same set of ﬁrst-order features for every related problem.
Since (cid:98) Ais also the same, our ﬁrst-order approximation has the same representation in all related problems.
A ﬁrst-order feature has to be fully grounded in order to evaluate its value in a state.
In contrastwith bound variables which are ground with objects in the arguments of actions, free variables canbe ground to any object of the same type.
We discuss the grounding of free variables in Section 4.3.
Service robot domain.
We introduce a novel domain, Service Robot ( SR ), in this paper.
A Gazebo environment for SR is illustrated in Figure 1.
In SR , a TIAGo robot has to assist somepeople.
The robot navigates from location pos 1 to pos 2 with the action move ( pos 1 , pos 2 ) .
A person can request assistance (represented by the state predicate need assistance ( person ) ) whichis a probabilistic exogenous event.
The robot does not initially know where the people are locatedand has to ﬁnd them with the action find ( person ) .
It then moves to the person and talks to them( talk ( person ) ).
If the person has requested assistance, the robot would receive some tasks (orgoals) of two possible types: (1) bring an item obj to location wp (represented by the state predicategoal object at ( obj, wp ) ) which requires the actions pick up ( obj, wp ) and put down ( obj, wp ) , and (2) deliver obj to person ( goal object with ( obj, person ) ) which requires the actiongive ( obj, person ) .
TIAGo can also take an item from a person ( take ( obj, person ) ).
Example 1.
In a ground approximation, we use the predicates ¯ P = { need assistance ( P 1 ) , ...,need assistance ( P m ) } , where P i are objects of the type person , as features (i.e., | P | numberof features).
In a ﬁrst-order approximation, these predicates are lifted to a ﬁrst-order feature. Forthe lifted action talk ( person ) , φ i = need assistance ( person ) . To evaluate φ i , it has to begrounded.
The bound variable person is substituted according to the variable binding of talk(e.g. , for talk ( P 3) , person is substituted with P 3 ). For move ( pos 1 , pos 2 ) , the ﬁrst-order feature isφ j = need assistance ( (cid:63)person ) ; because move does not have an argument of type person , P i issubstituted with a free variable (cid:63)person . φ j could be grounded by substituting (cid:63)person to any object of type person .
The number of ﬁrst-order features does not increase with the number of personobjects unlike the ground approximation.
This demonstrates that the ﬁrst-order approximation doesnot scale with the number of objects and is independent of O .
4.2 Online Feature Discovery
In Section 4.1, we describe the initialisation of Φ where each feature is a lifted state predicate.
Therecould be large approximation errors in (cid:101) Q as each of these feature has a high coverage. Hence, addingconjunctive state predicates, which have lower coverage, to Φ could reduce the approximation error.
The problem of feature discovery is to determine which features to add. We utilise iFDD+ , a modelfree, online feature discovery algorithm (line 6 of Algorithm 1).
A set of candidate features Φ caconsists of features which are the conjunction of any two features (its parent features ) in Φ a .
We denote the generation of Φ ca from Φ a as P ( Φ a ) .
iFDD+ iteratively adds candidate features toΦ a at each time step if their cumulative approximation errors, or relevances, exceed the discoverythreshold ξ .
The relevance of a candidate feature is updated at each time step by accumulating the
TD error only if the candidate feature is active.
A feature is active if it is true in the state and is nota parent feature of any active feature.
The weight of a newly added feature is equals to the sum ofthe weights of its parents.
When a feature φ i is added to Φ a , this in turn generates new candidatefeatures with φ i as one of its parent features.
4.3 Contextual Grounding of Free Variables
To evaluate the values of ﬁrst-order features in a state, they must be fully ground.
This is required in Algorithm 1 to determine the Q-values (lines 3 and 5) and the active features and candidate features(line 6).
The grounding inﬂuences the step update of w (line 5) and feature discovery (line 6). Bound variables are substituted with objects of the same type in the arguments of a ground action(see Example 1).
Features with at least one free variable are not fully ground.
The set of possiblesubstitutions for these features consists of every combination of objects for the free variables.
Thenumber of substitutions for a ﬁrst-order feature φ i and for Φ ˆ a are:
[Equation]
respectively, where σ φ i ( σ Φ ˆ a ) is the set of possible substitutions for φ i ( Φ ˆ a ), v is the set of freevariables in φ i and | v | is the number of objects of the same type as v .
If we consider every substitution (i.e., to treat a free variable like an existential quantiﬁer) where φ i is true in s if thereexists a substitution in σ φ i which makes it true, then there would be many such states (e.g., in
Example 1, need assistance ( (cid:63)person ) will be true if any person needs assistance).
Thus, theﬁrst-order approximation has a coarse granularity which can deteriorate performance.
We proposetwo methods to resolve this.
First, we use contextual knowledge to reduce the number of objects which a free variable can be grounded to (i.e., reduce | v | in Equation 1). Any substitutionwhich conﬂicts with the substitution due to contextual knowledge is removed from σ Φ ˆ a . Twosubstitutions conﬂict if they substitute a free variable with a different object.
We introduce twoforms of contextual knowledge which require trivial domain knowledge: goal and location.
Second, after contextual knowledge is applied, we select σ M ⊆ σ Φ ˆ a such that a metric M is maximised: σ M = { arg max σ ∈ σ Φ ˆ a M (cid:0) Φ σ ˆ a (cid:1) } where Φ σ ˆ a is the grounding of Φ ˆ a with σ .
We deﬁne
M (cid:0) Φ σ ˆ a (cid:1) = (cid:80) φ i ∈ Φ σ ˆ a φ i ( s, a )( | φ i | ) 2 where | φ i | is the number of state predicates in φ i .
This maximises the number of active, complex features.
The choice of M must be dependent on the stateonly to preserve the Markovian property of RMDPs.
Goal Context.
The goals G in a planning problem are state predicates which can be trivially determined from either the reward function or the deﬁnition of terminal states, both of which arecommonly assumed to be known.
The objects in g ∈ G are used for grounding free variables of thesame type: σ goal = g = { (cid:63)x/X, . .
, (cid:63)z/Z } where g = p ( X, . . .
, Z ) . We consider only active goals;a goal is active if it is known and has not been achieved (e.g., in SR , a goal is known only after aperson has given a task).
We assume that achieved goals are inconsequential for decision-makingin the present and future.
The Q-values after applying goal context represent the expected values ofactions for achieving the goal.
This is similar to the goal-associated Q-function in [25] but is notthe same as additive rewards as deﬁned in [26] where each goal is assumed to contribute uniformlyand additively to the reward.
φ i = {+
+} if there exists a g such that σ goal = g results in φ i being true; thevalue does not change even if more of such goals exist.
A goal typically gives a contextual grounding which is in conﬂict with another goal.
We proposetwo goal selection schemes: (1) GA uses every goal with the maximal metric M , and (2) G1 usesonly one goal at a time.
G1 selects the next goal in an ordering of active goals, ¯ G , when the currentlyselected goal is achieved.
The order in ¯ G could affect the optimality and soundness of the policy.
Indomains where there is a sequence to achieving goals, the choice of ¯ G is crucial.
We determine ¯ Gfrom the state trajectories observed in the previous episode.
A partial order g i <
g j is added if g iis achieved before g j .
¯ G is randomly shufﬂed at the start of an episode such that every partial orderis satisﬁed.
If every goal is achieved, then the total order is known though it might not be optimal.
However, due to exploration in RL, the order of goals achieved might not be the same as ¯ G .
Thus, amore optimal ¯ G could still be found.
Example 2.
In SR , the goals are represented by the state predicatesgoal object with ( obj, person ) and goal object at ( obj, wp ) .
Suppose that the goalsare to deliver item O 1 to location WP 5 and item O 2 to person P 1 . The former is denoted asg 1 = goal object at ( O 1 , WP 5) and the latter as g 2 = goal object with ( O 2 , P 1) .
The goalg 1 involves the objects O 1 with type obj and WP 5 with type wp , and the corresponding contextualgrounding is σ goal =
g 1 = { (cid:63)obj/O 1 , (cid:63)wp/WP 5 } . Likewise, σ goal = g 2 = { (cid:63)obj/O 2 , (cid:63)person/P 1 } .
Location Context.
Mobile robots move in an environment and can only interact with objects in theirvicinity.
Following this observation, location context substitutes the free variable with the currentlocation of the robot: σ loc = { (cid:63)wp/WP } if the robot is at location WP of type wp .
Locationcontext can be used with the goal context if there is no conﬂict (i.e., they substitute free variables ofdifferent types)—this queries if the robot is at the same location as the goal of interest.
If there is aconﬂict, then location context takes precedence.
Example 3.
In SR , the state predicate robot at ( wp ) represents the location of a robot.
If therobot is at WP 4 , then the contextual grounding is σ loc =
{ (cid:63)wp/WP 4 } .
Following Example2, we can apply σ goal =
g 2 and σ loc together without any conﬂict to give the contextual groundingσ loc,goal = g 2 = { (cid:63)wp/WP 4 , (cid:63)obj/O 2 , (cid:63)person/P 1 } .
On the other hand, σ goal = g 1 and σ loc areconﬂicting as they ground (cid:63)wp to WP 5 and to WP [-1 . However, they can still be applied together if σ loctake precedence: σ loc,goal = g 
 = { (cid:63)obj/O 2-]4 , respectively.
They can still be applied togetherby letting either one takes precedence (i.e., ignore the grounding of the other one).
If location contexttakes precedence, then σ loc,goal = g 1 = { (cid:63)obj/O 1 , (cid:63)wp/WP 4 } .
4.4 Granularity of First-Order Approximations
The lifted state-action space, (cid:98) S × (cid:98) A , is often much smaller than the ground state-action space—alifted predicate p ( x, y ) can be ground in | x
|| y | ways.
Since (cid:98) S , (cid:98) P , and (cid:98) A are independent of O , thelifted state-action space does not increase with the scale of the planning problem which makes ﬁrstorder approximations scalable to large-scale problems (see Example 1).
However, the downsideis a coarser granularity in approximation.
The maximum number of features is (cid:80) a ∈
A | Φ ( a ) | =2 2 | P | ×| A | in a ground approximation, and (cid:80) ˆ
a ∈ (cid:98) A | Φ ˆ a | = 2 2 | (cid:98) P | ×| (
cid:98) A | in a ﬁrst-order approximation.
Since | (cid:98) P | ≤ | P | and | (cid:98)
A | ≤ | A | (they are equal if there is one object for each type), a ﬁrst-orderapproximation uses fewer features to partition the state space.
Therefore, the size of the partitions(or granularity) must be larger than a ground approximation.
Furthermore, free variables in a ﬁrstorder feature increase its coverage because there only needs to exist a substitution amongst severalpossible substitutions to make the feature true.
This deteriorates performance if there are plateausin the policy induced.
Plateaus are regions of the state space where there are multiple non-optimalactions with the maximal Q-value.
The coarse granularity limits the type of problems a ﬁrst-order approximation can be applied in.
Because ﬁrst-order approximations do not consider every ground state predicates, it is suited for planning in factored MDPs with independent goals.
Experiments and Results
We demonstrate empirically that our work (1) learns a generalised and goal-directed policy, (2)enables transfer learning from small-scale problems ( P small ) to large-scale problems ( P large ) andbetween different simulation environments, and (3) reduces memory and computational costs.
Results are averaged over 10 independent runs where a different, randomised problem is used.
We used(cid:15) - greedy policy with a linearly decaying (cid:15) over episodes.
The parameters used are (cid:15) initial = 1 forexperiments without transfer learning and 0 .
otherwise, α = 0 .
3 (learning rate), γ = 0 .
9 , λ = 0 .
7(decay rate for eligibility traces), and ξ = 3 .
H is 20 ( 40 ) for P small ( P large ).
We assume that thepreconditions of actions are known (i.e., an illegal action will never be executed).
Experimental Setup.
In P small ( P large ), O contains three (six) items, one person (three people),and ﬁve (ten) locations and | S | × | A | = 2 56 × 65 ( 2 216 × 264 ); the probability of people needingassistance is 0.5 (0.5, 0.3, and 0.0 for three people).
Each person gives two tasks.
The problems arerandomised in the locations of items and in the tasks given.
Executing an action gives a reward of− 1 and completing a goal gives 20 .
We tested our work in two simulation environments: RDDLSim[27] and ROS (Robot Operating System) with the ROSPlan [28, 29] package.
Both simulators returnan observation for executing an action in a state (line 4 of Algorithm 1).
We augment the observationfrom RDDLSim with ∆ , approximated as the mean of the observed ∆ in ROS injected with normallydistributed noise (20% of a standard deviation).
We built a Gazebo environment (see Figure 1) fora more realistic simulation relative to RDDLSim .
Firstly, in RDDLSim , the robot loses localisationwhen moving with a probability of 0.1; in ROS , it loses localisation when the covariance of the poseestimation from adaptive Monte-Carlo localization exceeds a threshold.
Secondly, in RDDLSim , therobot always succeeds in ﬁnding a person and ends up in the same location as the person.
In ROS ,the robot follows an exploration path to ﬁnd a person.
It might ﬁnd the intended person and/orother people it encountered along the way, and might not be in the same location as the person sinceit could detect people from a distance.
Lastly, in RDDLSim , picking up and putting down an itemalways succeeds while these actions might fail in ROS .
Each experiment in RDDLSim is conductedon a single core of AMD Opteron 6376 and with at most 4 GB of RAM allowed.
Ablation Study.
We conducted an ablation study for contextual grounding in ﬁrst-order approximation and benchmark them against the ground approximation.
RDDLSim was used.
The resultsare shown in Figure 2.
We measure performance by the total undiscounted reward received in eachepisode, and by the total duration of plan execution (mission time).
Executing an action incurs a re- ward of − 1 which could result in a learned policy which minimises the number of actions executed;this seems to sufﬁce in reducing the mission time as it was observed that the mission time reducesasymptotically while the reward increases.
G1 and Loc.
+ G1 (combination of location context and G1 ) generally outperforms GA and Loc.
+ GA .
This is because the goals in SR are independent of eachother and considering one goal at a time reduces σ Φ ˆ a and coarseness of granularity more than GAdoes.
In P small , using no context or only location context has the worst performances.
Due toour modelling choice to include wp in arguments of actions (e.g., pick up ( obj, wp ) ), features with(cid:63)wp are unnecessary to approximate the Q-values of some actions—we opt against using domainknowledge to prune these features.
Nevertheless, location context is crucial in reducing the memoryand computational cost; experiments for P large cannot be completed without location context as ittakes too long.
The computation time for P large is further reduced when goal context is also used.
The computation time for P small are comparable for all runs and are thus omitted.
Although groundapproximation has the best performance in P small , it could not be used in P large as the memory usage exceeds the 4 GB RAM limit.
This highlights the poor scalability of ground approximation—for P small , the ground approximation has 6894 ± 2 .
7 (one standard deviation) features while the variousconﬁgurations of ﬁrst-order approximation have 280 to 316 features.
Transfer Learning.
In Figure 2, we show the results of transfer learning where (cid:101) Q trained in P small( (cid:101) Q P small ) is used, and continues to be updated, in P large .
The contextual grounding used in P smallmust also be used in P large for transfer learning to work.
We consider only Loc.
+ G1 or Loc.
+ GAas these are computationally tractable.
The former outperforms the latter for the aforementionedreasons.
The boost in performance due to transfer learning is evident in episode 1 .
The asymptoticperformance without transfer learning approaches that of transfer learning as expected.
However,for Loc.
+ GA , transfer learning performed asymptotically poorer than without transfer learning.
Thisis because GA considers multiple goals for contextual grounding.
Thus, it generalises poorly from
P small , which has two goals, to P large , which has four goals.
[Table]
Lastly, we evaluate the generalisation from
RDDLSim simulations to ROS simulations.
We used ten randomised problems for
P small and P large which are different fromthe ones used in RDDLSim .
A greedy policy induced by (cid:101) Q P small using Loc.
+ G1can directly solve P small but not P large .
Therefore, we trained (cid:101) Q P small further in500 episodes of P large using RDDLSim witha modiﬁed transition function (i.e., therobot’s location does not change when ﬁnding a person) to solve P large .
Without further training, the policy could fail to instructthe robot to move to a person after ﬁndingthat person.
The results are shown in Table 1 alongside the results from RDDLSim (i.e., Figure 2) forease of comparison.
For P small , the performance in RDDLSim is better with smaller variances whichis expected since the simulations do not consider real-world dynamics and are more optimistic.
Surprisingly, the results for P large are comparable for both simulation environments.
Thus, we havedemonstrated that ﬁrst-order approximations can generalise over different simulation environments.
Conclusion
We propose a model-free, online RRL algorithm which learns an approximation of the Q-functionwith ﬁrst-order features that generalises across different problems of the same domain.
Free variables in the ﬁrst-order features caused ambiguity in how they should be substituted.
The choice of asubstitution inﬂuences the learning and is critical to performance.
We resolve this problem by utilising contextual knowledge to substitute free variables.
Empirical results showed that our methodis scalable to large-scale problems by keeping memory usage and computational cost tractable, andcan generalise across problems of different scales and across different simulation environments.
Infuture work, we plan to investigate other forms of contextual knowledge and to consider temporalinformation such as duration or time constraints.