Learning Search-Space Speciﬁc Heuristics Using Neural Network
Abstract
We propose and evaluate a system which learns a neural- network heuristic function for forward search-based, satisﬁcing classical planning.
Our system learns distance-to-goal es- timators from scratch, given a single PDDL training instance.
Training data is generated by backward regression search or by backward search from given or guessed goal states.
In domains such as the 24-puzzle where all instances share the same search space, such heuristics can also be reused across all instances in the domain.
We show that this relatively sim- ple system can perform surprisingly well, sometimes competitive with well-known domain-independent heuristics.
Introduction
State-space search using heuristic search algorithms such as
GBFS is a state-of-the-art technique for satisﬁcing, domainindependent planning.
Search performance is largely determined by the heuristic evaluation function used to decide which state to expand next.
Heuristic function effectiveness for domain-independent planning depends on the domain, as different heuristics represent different approaches to exploiting available information.
Designing heuristics which work well across many domains is nontrivial, so learning-based approaches are an active area of research.
In one setting for learning search control knowledge for planning (exempliﬁed by the Learning Track of the IPC), a set of training problem instances (and/or a problem instance generator) is given and the task is to learn a domain-speciﬁc heuristic for that domain.
Previous work on learning heuristics and other search control policies (e.g., selection among several search strategies) in this setting include (Yoon, Fern, and Givan 2008; Xu, Fern, and Yoon 2009; de la Rosa et al.
; Garrett, Kaelbling, and Lozano-P´erez 2016; Sievers et al. 2019; Gomoluch, Alrajeh, and Russo 2019).
Another type of setting seeks to learn domain-independent planning heuristics, which generalize not only to domains used during training, but also to unseen domains (Shen, Trevizan, and Thi´ebaux 2020; Gomoluch et al. 2017).
Inter-instance speedup learning, or “on-line learning”, is a setting where only one problem instance (no training instances or problem generator) is given, and the task is to solve that instance as quickly as possible.
Speedup learning within a single problem solving episode is worthwhile if the total time spent by the solver (including learning) is faster than the time required to solve the problem using other methods.
Previous work on on-line learning for search-based planning includes learning decision rules for combining heuristics (Domshlak, Karpas, and Markovitch 2010) and macro operator learning (Coles and Smith 2007).
On-line learning can be used to learn an instance-speciﬁc heuristic .
Previous work on instance-speciﬁc learning includes bootstrap heuristic learning (Arfaee, Zilles, and Holte 2011), as well as LHFCP, a single-instance neural network heuristic learning system (Geissman 2015).
Instancespeciﬁc learning can be generalized to single search space learning , where many problem instances share a single search space.
For example, all instances of the 15-puzzle domain share the same search space – different instances have different initial states, all on the same connected state space graph.
Thus, a learned heuristic function which performs well for one instance of the 15-puzzle can be directly applied to other instances of the domain.
We propose and evaluate SING, a neural network-based instance-speciﬁc and single search space heuristic learning system for domain-independent, classical planning.
SING is closely related to LHFCP, an approach to supervised learning of heuristics which generates training data using backward search (2015).
Given a PDDL problem instance I , LHFCP learns a heuristic h nn for I .
To generate training data for h nn , LHFCP performs a series of backward searches from a goal state of I to collect a set of states and their approximate distances from the goal.
After training, h nn is used as the heuristic function by GBFS to solve I .
This does not require any additional training instances as input, nor preexisting heuristics to bootstrap its performance.
However,
LHFCP performed comparably to blind search (Geissman 2015), so achieving competitive performance with this approach remained an open problem.
SING expands upon this basic approach in several ways: (1) improved backward search space using either (a) explicit search with inferred inverse operators or (b) regression, (2) depth-ﬁrst search (vs. random walk), (3) boolean state representation, and (4) relative error loss function.
We experimentally evaluate SING for learning domain-speciﬁc heuristics for domains where instances share a single state space, and show performance competitive with the Fast Forward heuristic ( h ff ) (Hoffmann and Nebel 2001), and the land- mark count heuristic ( h lm ) (Hoffmann, Porteous, and Sebastia 2004) on several domains.
We also evaluate SING as an instance-speciﬁc heuristic learner, and show that the learned heuristics consistently outperforms blind search on a broad range of standard IPC benchmark domains, and performs competitively on some domains, even when the learning times are accounted for within the time limit.
Preliminaries and Background
We consider domain-independent classical planning problems which can be deﬁned as follows.
A SAS+ planning task (B¨ackstr¨om and Nebel 1995) is a 4-tuple, Π = (cid:104) V, O, I, G, (cid:105) , where V = x 1 , ..., x n is a set of state variables, each with an associated ﬁnite domain Dom ( x i ) ;
A state s is a complete assignment of values to variables.
S is the set of all states.
O is a set of actions, where each action a ∈ O is a tuple ( pre ( a ) , eﬀ ( a )) , where pre ( a ) and eﬀ ( a ) are sets of partial state variable assignments x i = v, v ∈ Dom ( x i ) ; I ∈ S is the initial state, and G is a partial assignment of state variables deﬁning a goal condition ( s ∈ S is a goal state if G ⊂ s ).
A plan for Π is a sequence of applicable actions which when applied to I results in a state which satisﬁes all goal conditions.
Searchbased planners seek a path from the start state to a goal state using a search algorithm such as best-ﬁrst search guided by a heuristic state evaluation function.
A natural approach to learn heuristic functions for searchbased planning is a supervised learning framework consisting of the following stages: (1) Training Sample Generation : generate many state/distance pairs which will be used as training data.
(2) Training : Train a heuristic function h which predicts distances from a given state to a goal.
Search : Use h as the heuristic evaluation function in a standard heuristic search algorithm such as GBFS.
This paper focuses mostly on stage (1), training data generation.
Ferber et al.
(2020) investigated an approach where the training data was generated using forward search from the start states of training instances.
They perform random walks (200 steps) from the initial state, and from each step visited in the random walk, they perform a forward search (a “teacher search” using a heuristic such as h ff ) to a goal in order to ﬁnd the distance to the goal.
If the teacher search ﬁnds a path to the goal, the states on the path as well as the distance-to-goal for the states on the path are added to the training data.
This approach can be practical for shared search space heuristic learning, where the costs of the teacher searches can be amortized among many instances on the same search space.
However, this is not practical for satisﬁcing, single instance heuristic learning where there is only one problem solving episode, as requiring forward search to the goal in order to gather training data obviates the need to learn a heuristic for that particular instance.
An alternative approach to generating training data uses backward search from the goal.
A backward search starting at a goal state (provided in or guessed/derived from the problem speciﬁcation) is performed, storing encountered states and their (estimated) distances from thoe goal as the training data.
Arfaee, Zilles, and Holte (2011) used this approach in a bootstrap system for heuristic learning, which starts with a weak neural net heuristic h 0 and generates increasingly more powerful heuristics by using the current heuristic to solve problem instances, using states generated during the search as training data for the next heuristic improvement step.
If h 0 is too weak to solve training set problems, they generate training data by random walks from the goal state to generate easy problem instances that can be solved by h 0 .
Lelis et al.
(2016) proposed BiSS-h, an improvement which uses a solution cost estimator instead of search for training data generation.
Arfaee et al.
and Lelis et al. evaluated their work on domain-speciﬁc solvers for the 24-puzzle, pancake puzzle, and the Rubik’s cube.
Geissmann (2015) investigated a backward-search approach to training data generation for domain-independent classical planning.
His system, LHFCP, uses backward search to generate training data for learning a neural network heuristic function which estimates the distance from a state to a goal.
To generate training data, LHFCP performs backward search (random walk) in an explicit search space.
It generates the start state for backward search by generating a state which satisﬁes the goal conditions, with values unspeciﬁed by the goal condition ﬁlled in randomly.
LHFCP relies on the operators in the original (forward) problem to perform backward search.
Search using the heuristics learned by LHFCP across a wide range of IPC domains performed comparably to blind search (Geissman 2015).
Geissmann also investigated a variation of LHFCP which applied BiSSh to classical planning but reported poor results, attributed to difﬁculties in efﬁciently implementing BiSS for classical planning.
Thus, a successful backward-search based approach to training data generation for domain-independent classical planning remained an open problem.
SING: An Improved, Backward-Search Based Heuristic Learning System
We describe SI ngle search space N eural heuristic G enerator (SING), a system which learns single-search space heuristics for domain-independent planning.
SING learns a heuristic function h nn ( s ) , which takes as input a vector representation of a state s , and returns a heuristic estimate of the distance from s to a closest goal.
SING is implemented on top of the Fast Downward planner (FD) (Helmert 2006).
SING uses backward search to generate training data, similar to LHFCP, but incorporates several signiﬁcant differences in the state representation, backward search space formulation, and backward search strategy.
Below, we describe each of these in details:
3. State Representation
The input to h nn is a vector representation of a state.
LHFCP used a multivalued SAS+ vector representation of the state, which is a natural representation to use, as FD uses the SAS+ representation internally.
Another natural representation for the vector input to h nn is based on the STRIPS propositional representation of the problem.
A STRIPS planning task (Fikes and Nilsson 1971) is a 4-tuple, Π = (cid:104) F, I, G, A, (cid:105) where F is a set of propositional facts, I ∈ 2 F is the initial state, G ∈ 2 F is a set of goal facts, and A is a set of actions.
Each action a ∈ A has preconditions pre ( a ) , add effects add ( a ) , and delete effects del ( a ) , which are sets of facts.
A state s ∈ 2 F is set of facts, and s is a goal state if G ⊆ s .
Given a state s ∈ 2 F , a is applicable iff pre ( s ) ⊆
s .
After applying a in s , s transitions to s ∪ add ( a ) \ del ( a ) .
A plan for Π is a sequence of applicable actions which make I transition to a goal state.
The STRIPS representation corresponds directly to the classical planning subset of the standard PDDL domain description language, as PDDL uses boolean facts to represent the world state.
In the SAS+ representation used by FD, each possible value of a variable represents a mutually exclusive set of facts in the underlying propositional problem.
Each variable-value pair in FD represents a fact, negation of a fact, or negation of all facts represented by other values in the variable.
Preconditions and effects of actions are also represented as the set of variable-value pairs.
Since the variable/value naming conventions used in the SAS+ generated by the FD PDDL-to-SAS+ translator, conversion between the SAS+ ﬁnite-domain representation and the STRIPS propositional state representation is easy.
Thus, h nn can use either the boolean (STRIPS) or multivalued (SAS+) state vector representation as input during training and search.
Since each input bit corresponds to a fact in the boolean encoding, it may enable a more accurate h nn state evaluation function to be learned than the SAS+ multi-valued encoding.
On the other hand, SAS+ encodings are more compact, which can signiﬁcantly reduce the dimensionality of the state representation, which can result in faster NN evaluation, speeding up the search process.
Thus, the choice of state vector representation poses a tradeoff between h nn evaluation accuracy and h nn evaluation speed, and SING can use either the multivalued SAS+ vector representation or the
STRIPS boolean vector representation.
3. Search Space and Operators for Training Sample Generation
The task of training sample generation is to collect a training set T = { ( s 1 , e 1 ) , ..., ( s r , e r ) } a set of r states and their (estimated) distance to a goal.
The basic idea is to repeatedly start at a goal g and generate a sequence of states heading away from it (using a directed search or random walk), adding such states to the training data.
In some search problems such as the sliding tiles puzzle, backward search is relatively straightforward as the goal state is given explicitly as input to the problem, and the operators available for the forward problem are sufﬁcient to solve the backward problem.
In domain-independent planning, backward search based training sample generation poses several issues.
First, a goal condition, possibly satisﬁed by many goal states, is given instead of an explicit, unique goal state, so in general, it is not possible to simply “search backward from the goal state”.
Second, in general, the operators for the forward problem are not sufﬁcient for backward search.
LHFCP generates a start state for backward search by generating a state which satisﬁes the goal conditions, with values unspeciﬁed by the goal condition ﬁlled in randomly.
It relies on the operators in the original (forward) problem to perform backward search.
SING incorporates two approaches to backward search for training sample generation: (1) backward explicit search using derived inverse operators, and (2) regression.
Explicit Backward Search with Derived Inverse Operators As in LHFCP, a candidate start state for backward search is generated by ﬁrst generating a partial state which satisﬁes all conditions in the goal condition, and then randomly assigning values to variables whose values are unspeciﬁed in the goal condition.
Such a randomly generated candidate start state s might be invalid and unexpandable, i.e., no backward operators (see below) can be applied to s .
In that case, we simply generate another candidate state.
This random initialization is performed for each backward sampling search.
For the search operators, one simple approach is to use the same set of actions as for forward search, as in LHFCP (Geissman 2015).
However, this fails in domains where actions are not invertible such as visitall .
Thus, operators for the backward search must be derived from the forward operators.
Since preconditions and effects are represented as a set of variable-value pairs in FD, one naive method to generate inverse actions is to swap values of variables which appear in both preconditions and effects.
Other variable-value pairs in preconditions and effects are treated as preconditions in the inverse action, because they must hold after application of the action.
However, the inverse action does not change values of variables which appear in the original effects but not in the original preconditions.
To address this issue, we use information available in the STRIPS formulation of the problem (as explained in Section 3.1, conversion among the PDDL problem description, its STRIPS formulation and the SAS+ formulation used internally by Fast Downward is straightforward).
For action a , we generate an inverse action a (cid:48) such that pre ( a (cid:48) )
=
( pre ( a ) ∪ add ( a )) \ del ( a ) , add ( a (cid:48) )
=
del ( a ) , and del ( a (cid:48) )
= add ( a ) .
We identify variable-value pairs which represent propositions as add effects, and pairs which represent negation of facts as delete effects.
Regression Another approach to backward search is regression.
In backward search using regression, we use the modiﬁed SAS+ representation by Alc ´ azar et al.
An action a is applicable to a state s if add ( a ) ∩ s (cid:54) = ∅∧ del ( a ) ∩ s = ∅ .
If an action a is applied to a state s , s transitions to a state s (cid:48)
=
( s \ add ( a )) ∪ pre ( a ) .
Normally, SAS+ variables represent mutex groups of the corresponding STRIPS propositions.
In regression planning with SAS+, each variable has an additional, undeﬁned value.
The starting node in regression space is the goal state, where variables unspeciﬁed by the goal condition have undeﬁned values.
When an action a is applied to a state s , if a variable v is included in add ( a ) but not in pre ( a ) , v is set to undeﬁned .
When generating training data, a bit vector representation of states needs to be generated (Section 3.1).
When converting the SAS+-based representation used by Fast Downward into a bit vector, unlike the other possible state values in the mutex group, undeﬁned values are not explicitly represented in the bit vector.
For example, suppose a state variable x in regression search has 2 possible actual values, v 1 and v 2 , as well as ”undeﬁned”.
In the bit vector representation output for use as training data, x is represented by 2-bits, where the ﬁrst bit represents x 1 , and the second bit represents x 2 , and there is no explicit third bit for the undeﬁned value.
Regression vs. explicit search spaces The choice of regression vs. explicit spaces depends on the domain.
Although regression is in a sense the “correct” way to perform backward search, the backward branching factor in regression space is very large in many domains.
On the other hand, while explicit backward spaces sometimes have much smaller branching factor than regression, goal generation has risks.
First, goal generation might fail to ﬁnd a goal.
Also, generated goals might not be true goal states reachable from the start state, and states reachable from such incorrect states are also unreachable from the start state.
Such cliques (unreachable from the start state) can cause backward search to yield few or no states for training data.
However, although the training data may include states which are unreachable from the start state, these may nevertheless be useful for learning an effective h nn which evaluates “real” states during search, somewhat similar to how synthetic data generated by the adversary during training is useful for learning networks which correctly classify real data in GAN learning (Goodfellow et al. 2014).
3. Backward Search Strategy
Given a start state for backward search (corresponding to a goal in the forward search space), we seek a set of training states T which are relatively far from goal g but with a reasonable estimate of their distance from g for training h nn .
Breadth-ﬁrst search (BFS) from g could be used to generate states T for which c ( s, g ) , the exact distances from s ∈ S to g (assuming unit-cost domains) are known, but would limit the training data to states which are very close to g .
We need a search algorithm which can go much further from g than
BFS, and for which the number of steps in the (inverted) path from s ∈ T to g is an approximation of c ( s, g ) .
One natural sampling/search strategy is random walk, as in LHFCP (Geissman 2015).
The number of steps from g at which s is encountered is used as an estimate of the true distance from g to s .
Although random walk is fast, distance estimates from random walk may be inaccurate if cycles are not detected.
Loop detection can be implemented easily using a hash table, but in domains with many cycles, it can be difﬁcult to sample nodes far from g if the random walk is restarted whenever a previously visited node is generated.
Therefore, we use depth-ﬁrst search (DFS) to extend a path from g , using the depth at which s is encountered is used as an estimate of the true distance from g to s , and all generated states are added to T .
Random tie-breaking among s is used to select the nodes among successors of s , Succ ( s ) , to expand.
A hash table is used to prune duplicate nodes and prevent cycles.
In domains with many cycles and dead ends, by backtracking (instead of restarting search) when a duplicate is detected, DFS can potentially sample more states which are further from g than random walk.
The best choice of sampling search strategy depends on the domain.
In some domains, DFS generates more accurate samples than random walk due to duplicate detection and backtracking, while in other domains DFS may incur large overheads due to backtracking and Random walk allows faster searches.
In the experiments below, during training data generation we perform nsearches backward searches, stopping each search after nsamples states are collected, i.e., nsearches × nsamples states are collected.
3. Neural Network Architecture
We use a standard feedforward network for h nn , using the
ReLU activation function.
Each layer is fully connected to the next layer.
The input layer takes the state vector representing a state s as input.
As discussed in Section 3.1, the state vector is either a boolean vector for the STRIPS representation of the problem instance, or a multivalued vector for the SAS+ representation of the instance, so the number of inputs is the same as the length of the state vector ( | F | for STRIPS propositional representation, | V | for SAS+ multivalued representation).
The output layer is a single node which returns h nn ( s ) , the heuristic evaluation value of state s .
Since h nn will be called many times as the heuristic evaluation function for best-ﬁrst search, a small network enabling fast evaluation is desirable.
PyTorch 1.2.0 is used for training h nn , but for search, we use the Microsoft ONNX Runtime 0.4.0 to evaluate h nn .
Both training and search use a single CPU core.
Due to the simple network architecture as well as accelerated evaluation using the ONNX Runtime, h nn can be evaluated relatively quickly, signiﬁcantly faster than h ff on most IPC domains, (see node expansion rates in Table 2).
3. Loss Function
Previous work on learning neural nets for classical planning used the standard Mean Square Error (MSE) regression loss function (Geissman 2015; Ferber, Helmert, and Hoffmann 2020).
Instead of MSE, we use a prediction relative error sum loss function, f loss = (cid:80) i abs (ˆ y i − y i ) /
( y i + 1) , which is the sum of the relative error of the predicted ( ˆ y i ) values compared to the training data ( y i ).
Evaluation: Domain-Speciﬁc Heuristic Learning on Shared Search Spaces
In domains where multiple instances share the same space, it is possible to learn reusable h nn networks that can be used across many instances, so the cost of learning a heuristic can be amortized across instances.
For example, all instances of the N -puzzle (for a particular value of N ) share the same search space.
We evaluated SING as a shared search space, single model learner on the following PDDL domains:
• 24-puzzle : PDDL encodings of the standard 50instance benchmark set from (Korf and Felner 2002) • 35-puzzle : 50 randomly generated instances
name state backward rev.
inversion NN # of NN nodes samples # vector space search hidden hidden
C boolean regression
DFS yes
C
SAS+ explicit rand.
walk yes
C boolean explicit
DFS yes
C boolean explicit
DFS yes 16 4 ×
SING/L
SAS+ explicit rand.
walk no  10 backward search.
“NN # of hidden”: # of hidden nodes in h nn .
“NN nodes hidden”:
# of nodes per hidden layer.
“samples #” : # of sample states collected in the training data collection phase using the sampling search.
C2, C3 and C perform 500 searches, with a limit of 200 samples/search ( 10 5 samples).
C5 performs 800 searches, with a limit of 500 samples/search ( 4 × 10 5 samples).
• blocks25 : 100 blocks instances with 25 blocks gener-
ated
• pancake : 100 randomly generated instances with 14 pancakes.
For each domain above, we ran the learning phase (training data generation and h nn training) once to learn a heuristic h nn for the domain.
For 24-puzzle , we used the C4 conﬁguration (Table 1 shows conﬁguration details), and training data generation took 7 seconds and training took 61 seconds.
For blocks25 , we used the C5 conﬁguration, training data generation took 502 seconds and training took 228 seconds.
For pancake , we used the C4 conﬁguration, training data generation took 21 seconds and training took 377 seconds.
Note that for these 3 domains, we tried several SING conﬁgurations (i.e., manual tuning) and report the results for the best conﬁguration.
We are currently investigating automated tuning (hyperparameter optimization) to optimize the best conﬁguration for a given domain.
Table 2 and Figures 1-2 compare the coverage, node expansions, and runtime (on solved instances) of GBFS using h nn , h ff , h lm with a 30 min time limit per instance and 8GB
RAM limit using an Intel(R) Xeon(R) CPU E5-2680 v2.
h nn had or tied for the highest coverage on all 4 domains.
On blocks25 and pancake , h nn had the highest coverage.
On 24-puzzle , 35-puzzle and pancake , h nn had the lowest median run time.
Thus, h nn achieved competitive performance on all of these domains compared to both h ff and h lm in this shared search space evaluation setting.
Note that while h nn and h ff expanded a comparable number of nodes, h nn had a signiﬁcantly higher median node expansion rate than h ff resulting in faster runtimes.
Figure 3 compares heuristic accuracy ( h -value minus true distance) for a set of 4400 states for h nn , h ff , h lm , and h gc (goal count).
For states with true distance ≤ 30 from the goal state, h nn is fairly accurate.
This accuracy and the fast evaluation speed due to the simple neural network enables efﬁcient, fast search.
Evaluation: Instance-Speciﬁc Learning
In Section 4, we evaluated SING for learning domainspeciﬁc heuristics which could be reused on many instances sharing the same search space, so the evaluation focused on search time, assuming that the time spent learning h nn can be amortized across multiple instances.
Next, we evaluate SING as an instance-speciﬁc learner in an IPC Satisﬁcing track setting, where SING is givenminutes total for all phases, including learning (including training data collection and training) and search.
Each run of SING starts from scratch – nothing is reused across instances, learning costs are not amortized, and the heuristic is learned speciﬁcally for solving a given instance once .
We evaluate SING on a large set of standard benchmarks from the IPC, all with unit-cost actions.
All runs were given a total 30 minutes for time limit both learning and search (i.e., includes training data collection, training, and search using h nn ) and 8GB RAM per instance.
We evaluated the SING conﬁgurations in Table 1.
As baselines for comparison, we also evaluated blind search, the goal count heuristic ( h gc ), the Fast Forward heuristic ( h ff ) (Hoffmann and Nebel 2001), and the landmark count heuristic ( h lm ) (Hoffmann, Porteous, and Sebastia 2004).
As an additional baseline we also evaluate SING/L, a conﬁguration of SING which is very similar to LHFCP (Geissman 2015) (see Table 1.
This conﬁguration is the same as C3, except that instead of the derived inverse operators (Section 3.2), SING/L uses only the actions available in the forward model.
Table 3 shows the coverage results (# of instances solved).
SING conﬁgurations C2, C3, C4, C5 signiﬁcantly outperform blind search, showing that SING successfully learned some useful heuristic information.
The SING/L (LHFCP) conﬁguration performed comparably to blind search, consistent with the results in (Geissman 2015).
Conﬁguration C3, which differs from SING/L only in that action inversion is used, has much higher coverage than SING/L, showing the effectiveness of action inversion.
C2 outperforms h ff on 5 domains and outperforms h lm on 2 domains.
C3 outperforms h ff on 5 domains and h lm on 1 domains.
C4 outperforms h ff on 4 domains, and C outperforms h ff on 3 domains.
Thus although none of the
SING conﬁgurations are competitive with h ff and h lm with respect to overall coverage, these results indicate that there are some domains where competitive performance can be obtained with a 30 minute limit, including the time to learn an instance-speciﬁc heuristic function entirely from scratch without a teacher.
Ablation Study
To understand the relative impact of each of the new components of SING compared to LHFCP, we performed an ablation study comparing the following conﬁgurations: (1) C5’:
Conﬁguration C5 (Table 1) with fewer training samples (100k instead of 400k), (2) C5’/rw : same as C5’, except using random walk instead of DFS, (3) C5’/sas : same as C5’, except using SAS+ instead of boolean state representation, (4) C5’/reg : same as C5’, except using regression instead of explicit search state, (5) C5’/orig : same as C5’,
[Table]
[Figure] [Figure]
[Figure]
u r
i s t
i c v a l u e - e x a c t d i s t a n c e h nn h ff h lm
[Figure]
except using original operators only (no action inversion), and (6) C5’/mse : same as C5’, except using MSE instead of the relative error sum loss function for NN training.
All conﬁgurations were run with a 30min, 2GB limit on the same IPC instances used in the above experiment.
The coverages of the conﬁgurations were 604 for C5’, 563 for
C5’/rw, 481 for C5’/sas, 651 for C5’/reg, 420 for C5’/orig, and 559 for C5’/mse.
This shows that the use of DFS in backward search, the use of boolean state representation, the use of action inversion, and the use of relative error sum loss function all have a signiﬁcant positive impact on performance.
On the other hand, the effect of using regression vs explicit state search for the backward search during training data generation is highly domain-dependent, with regression performing better on some domains and explicit search on others, as can be seen by comparing conﬁgurations C (which is the same as C5’/reg) vs. C5 in Table 3.
Related Work
A broad survey of learning for domain-independent planning is (Celorrio et al. 2012).
Satzger and Kramer (2013) developed a neural network based, domain-speciﬁc heuristic for classical planning.
They used random problem generators to create instances for training the neural network.
Their training process also relies on the use of an oracle (the
FD planner with an admissible heuristic) to provide true distance from a state to a goal.
Shen et al.
(2020) proposed an approach to learning domain-independent (as well as domain-dependent) heuristics using Hypergraph Networks.
They showed that it was possible to successfully learn domain-independent heuristics which performed well even on domains which were not in the training data.
As this approach uses a hypergraph based on the delete relaxation of the original planning instance, it is quite different from the minimalist approach taken in SING, which does not use any such derived features and uses only the raw state vector.
The training data generation method is forward search based, similar to the forward approach of Ferber et al.
described in Section 2 (Ferber, Helmert, and Hoffmann 2020).
In addition, while their work focuses on generalization capability and search efﬁciency (node expansions) across domains, with runtime competitiveness left as future work, our work seeks to achieve runtime competitiveness using a simple NN architecture.
Random-walk sampling of the search space of determin- istic planning problems for the purpose of learning a control policy for a reactive agent was proposed in (Fern, Yoon, and
Givan 2004).
This differs from SING in that SING learns a heuristic function which estimates distances to a goal state and and guide search (GBFS), instead of a reactive policy.
There is also a rapidly growing body of work on learning neural network based policies for probabilistic domains (c.f., (Toyer et al. 2018; Issakkimuthu, Fern, and Tadepalli 2018;
Groshev et al.
2018; Bajpai, Garg, and Mausam
; Garg,
Bajpai, and Mausam 2019)), which is also related to learning heuristic evaluation functions for deterministic domains.
Conclusion
We investigated a supervised learning approach to learning a heuristic evaluation for search-based, domain-independent classical planning, where the training data is generated using backward search.
Although LHFCP, a previous system, followed the same basic approach, it was performed comparably to blind search.
SING pushes this approach much further using (1) backward search for training data generation using regression, as well as derived inverse operator for explicit space search, (2) DFS-based backward search for training data generation, (3) a propositional input vector representation, and (4) a relative error loss function.
We showed that SING can achieve performance competitive with h ff and h lm on several domains, both in shared search space scenarios where heuristics can be reused across domains, as well as single-instance learning where both learning and search using the learned heuristic must be performed within a given time limit.
SING is a relatively simple, minimalist system.
SING uses only a PDDL description of a single problem instance as input.
No additional problem generators or training instances are used.
Learning is from scratch, and unlike the forward search based training data generation approach investigated by (Ferber, Helmert, and Hoffmann 2020), SING does not use any standard heuristics during training data generation.
It uses a very simple feedforward neural network architecture, with no feature engineering.
The only “features” used by SING are the raw state vectors.
SING does not exploit any structures used by standard classical planning heuristics such as delete relaxations and causal graphs in either the learning or the search phases.
Previous work used features derived/extracted from humandeveloped heuristics such as h ff and explored how learning could be used to exploit such features in new ways (Yoon, Fern, and Givan 2008; Xu, Fern, and Yoon 2009; de la Rosa et al. 2011; Garrett, Kaelbling, and Lozano-P´erez 2016; Shen, Trevizan, and Thi´ebaux 2020).
By pushing the performance envelope for a more minimal approach our results provide a baseline for future work on heuristic learning using more sophisticated features and methods.
As discussed in Section 3.2, explicit backward search (as opposed to regression) for training data generation can generate states which are not reachable from the start state.
Nevertheless, our results show that SING conﬁgurations which use explicit backward search perform quite well on some domains.
In future work, we will investigate in detail how unreachable states in the training data affect the quality of the learned heuristic and the performance of the (forward) search using the learned heuristic.