Reﬁnements on the Complementary PDB Construction Mechanism Anonymous
Abstract
Pattern database (PDB) is one of the most popular automated heuristic generation techniques.
A PDB maps states in a planning task to abstract states by considering a subset of vari- ables and stores their optimal costs to the abstract goal in a look up table.
As the result of the progress made on sym- bolic search over recent years, symbolic-PDB-based planners achieved impressive results in the International Plan- ning Competition (IPC) 2018.
Among them, Complementary 1 (CPC1) tied as the second best planners and the best non- portfolio planners in the cost optimal track, only 2 tasks behind the winner.
It uses a combination of different pattern generation algorithms to construct PDBs that are complementary to existing ones.
As shown in the post contest ex- periments, there is room for improvement.
In this paper, we would like to present our work on reﬁning the PDB construc- tion mechanism of CPC1.
By testing on IPC 2018 benchmarks, the results show that a signiﬁcant improvement is made on our modiﬁed planner over the original version.
Introduction
Over past few decades, there has been research on various automated heuristic generation techniques, including abstractions (e.g. PDBs), delete relaxations (e.g. h max ) and landmarks (e.g. LM-cut) (Helmert and Domshlak 2009).
Pattern database (PDB) is a memory-based heuristic generation technique that requires a pre-processing phase.
It maps original states to abstract states by considering a subset of variables, namely a pattern and stores their optimal costs to the abstract goal in a look up table.
Early experiments on
PDB heuristic search involve optimally solving the slidingtile puzzles (Culberson and Schaeffer 1998) and the Rubik’s cube (Korf 1997).
The automated PDB generation process dates back to the work of Edelkamp (2001).
Studies have shown that the heuristic could be improved by combining multiple PDBs.
Felner, Korf, and Hanan (2004) show that the heuristic from the addition of multiple disjoint PDBs is admissible.
On top of that, Holte et al.
(2006) suggest taking the maximum heuristic over different additive PDBs.
In addition, different representations for PDBs have been proposed.
Explicit PDBs are structured as hash tables and usually limited by memory, while symbolic PDBs (Edelkamp 2002) are structured as binary decision diagrams (BDDs) (Bryant 1986).
The succinct representation of state sets in
BDDs usually reduces the memory consumption and enables the construction of much larger PDBs.
The space of possible abstractions is huge due to exponentially many patterns and various cost-partitioning strategies.
In order to construct PDBs with good quality, several algorithms have been proposed, including genetic algorithm (Edelkamp 2006), bin-packing (Edelkamp 2006;
Franco et al. 2017) and hill climbing (Haslum et al.
Kissmann and Edelkamp 2011).
Also, some PDB evaluation approaches have been suggested, such as average heuristic value (Edelkamp 2006), random walk sampling (Haslum et al. 2007) and stratiﬁed sampling (Lelis et al. 2016;
Barley, Franco, and Riddle 2014).
The basic idea of the
Complementary PDB construction (CPC) mechanism is to keep generating PDB collections using a combination of the above-mentioned algorithms and add a new collection
P sel to the collection set S cur if S cur ∪ {P sel } is predicted to improve the heuristic.
Moreover, the parameters of the
PDB construction algorithms are dynamically adjusted in the course.
The results of IPC 2018 have conﬁrmed the effectiveness the CPC mechanism.
Both Complementary 1 (CPC1) and Complementary 2 (CPC2) planners solved 124 planning tasks and tied as the runner ups, only 2 tasks behind the winner, and they were the best non-portfolio planners.
Nonetheless, some ill behaviors of CPC1 have been revealed in the post contest experiments (Franco et al. 2018).
For example, using a single PDB construction algorithm, GAMERStyle (Kissmann and Edelkamp 2011), can solve substantially more tasks than using a combination of algorithms, which indicates that these algorithms are not well integrated.
Hence, the purpose of our research is to reﬁne the PDB construction mechanism of CPC1 by optimizing the algorithms and the evaluator as well as better integrating all components.
In this paper, the deﬁnitions of related concepts will be introduced.
Then, our reﬁnements on the pattern construction algorithms, the evaluator and the overall process will be presented.
Finally, the results from the experiments on IPC 2018 benchmarks will be summarized to demonstrate the effectiveness of our reﬁnements.
Background SAS+ Planning
Planning tasks are encoded into the SAS+ model on the
Fast Downward planning system (Helmert 2006).
An SAS+ planning task is deﬁned by a 4-tuple Π =
(cid:104)V , O , s 0 , s ∗ (cid:105) (B¨ackstr¨om and Nebel 1995).
V is a ﬁnite set of state variables, where each variable v ∈ V has as ﬁnite domain D v .
A full state is an assignment to all state variables, while a partial state is an assignment to a subset of state variables.
O is a ﬁnite set of operators, where each operator o ∈ O is deﬁned by a tuple (cid:104) pre o , eff
o , cost o (cid:105) , specifying the preconditions, the effects and the non-negative cost.
Both pre o and eff o are partial states.
An operator o is applicable to a state s if and only if s agrees on the values of all variables in pre o .
The result of applying o to s is a new state s (cid:48) that agrees on the values of all variables in eff o with the other variables unchanged.
s 0 is the initial state and s ∗ is the partial goal state.
A solution to a planning task is a deﬁned sequence of operators ( o 1 , . . .
, o n ) that leads from s 0 to a goal state that satisﬁes
s ∗ , i.e. o n ( . . .
o 1 ( s 0 ) . . . )
[ v ] =
s ∗ [ v ] for all v ∈ V ar ( s ∗ ) .
The task of cost optimal planning is to ﬁnd a solution with the least cost.
The causal graph of an SAS+ planning task is a directed graph ( V, E ) with V = V and ( u, v )
∈
E if and only if there is an operator
o ∈ O
such that eff o
( v ) is deﬁned and either pre o ( u ) or eff o ( u ) is deﬁned, u (cid:54)
= v (Helmert 2004).
An arc ( u, v ) in the causal graph implies that the change of v is dependent on the current assignment of u .
Pattern Databases
A pattern database (PDB) is a look up table of abstract states that stores their optimal costs to the goal.
It abstracts the problem space by mapping it to a subset of state variables P ⊆ V , i.e. a pattern, and removing from the preconditions and effects of operators, s 0 and s ∗ any variable not in P .
This kind of abstraction is said to be homomorphic as any transition ( s, s (cid:48) ) in the original problem space remains valid, thereby yielding admissible and consistent heuristic.
A PDB is constructed by a blind regression search from the abstract goal states (Culberson and Schaeffer 1998;
Edelkamp 2001), usually up to a time and memory limit.
If the limit is reached and the search is stopped at the depth d , a partial PDB will be formed and a heuristic value of d plus the minimum operator cost will be returned for any unvisited abstract state (Anderson, Holte, and Schaeffer 2007).
The size of a PDB is deﬁned as Π v ∈P |D v | , which is the cross product of the domain size of all variables in P .
A combination of multiple PDBs generally produces better heuristic, e.g. through addition (Felner, Korf, and Hanan 2004) and
maximization (Holte et al. 2006).
Multiple PDBs are additive if any operator o ∈ O affects at most only one
PDB, where o affects a PDB P if V ar ( eff o ) ∩ P (cid:54) = ∅ .
Given a PDB collection, the additivity can be ensured with cost-partitioning, which splits the operator costs among the
PDBs.
A simple approach adopted in the Complementary planners, zero-one cost- partitioning, is to set the cost of an operator o to zero if it has affected any PDB in the collection, and keep the original cost otherwise (Edelkamp 2001).
There are alternatives that could yield more informative heuristic, such as saturated cost-partitioning (Seipp et al.
, but they are usually more computationally expensive.
All the PDB collections are combined with the canonical function (Haslum et al. 2007) in the Complementary planners, which takes the maximum heuristic value over all admissible combinations.
Symbolic Search and PDBs
Symbolic search has gained tremendous popularity in cost optimal planning because of signiﬁcant memory saving and faster computation.
A state vector is a binary encoding of the variables and its length is given by (cid:80) v ∈V (cid:100) log |D v |(cid:101) .
A characteristic function is used to represent a set of states that returns true if and only if an encoded state is in the set.
In symbolic search, any characteristic function is succinctly represented in a binary decision diagram (BBD) (Bryant 1986).
In addition, any operator can be represented as a set of transition relations T c ( s, s (cid:48) ) in a BBD, where c is the operator cost, s is the predecessor state and s (cid:48) is the successor state.
The set of successors or predecessors for a state set can be generated via the image or the preimage operation, which takes the relational product of the state set and the transition relations (Torralba et al. 2017).
The size of a BBD depends on the ordering of the variables.
A local search algorithm has been proposed by Kissmann and Edelkamp (2011) that places related variables in the causal graph as close as possible, which performs much better than random ordering.
Symbolic A* search (BBDA*) partitions the open list into a matrix of g and h value where each bucket is represented by a BBD (Edelkamp and Reffel 1998; Jensen, Bryant, and
Veloso 2002).
It expands the buckets along an f diagonal each time, starting from the lowest g value, and adds the successor state sets, to avoid duplicate expansion.
Note that in the presence of zero-cost operators, additional layers in a bucket are needed to keep track of the blind search using only those operators.
Symbolic PDBs (Edelkamp 2002) can be constructed in a way similar to explicit PDBs.
A regression search is made from the set of abstract goals, and the sets of expanded abstract states are encoded in BBDs along with their costs to goal.
It is convenient to query on a heuristic value for a state set by doing a conjunction with the
BBDs, producing a subset of states with the heuristic value.
The Complementary PDB Construction Mechanism GAMER-Style
GAMER-Style (Kissmann and Edelkamp 2011) is a hill climbing algorithm for constructing a single large PDB.
The sketch is given in Algorithm 1.
It starts with all goal variables and sequentially adds a set of causally related variables that are predicted to improve the heuristic.
If the set of selected variables is empty, GAMER-Style will either terminate if the candidate variable set is also empty, or continue with the remaining candidate variables in the next call, i.e. a partial GAMER run.
In our implementation, the remaining candidate variables are shufﬂed before the partial run to avoid the effect of ordering (line 6).
The performance of GAMER-Style depends a lot on the evaluator E .
Although originally average h is used, random walk sampling is suggested by Franco and Torralba (2019).
Compared to average h, random walk sampling gives more precise estimate because the problem space distribution is approximated and known dead ends are removed (Haslum et al. 2007).
In light of this, our evaluator samples separately for GAMER-Style according to the initial h of P sel .
Resampling will be performed if and only if the initial h of the new
PDB increases by over 10% (line 16).
More details on the evaluator are explained in the Evaluator section.
Input : Planning task Π , evaluator E , time limit T and memory limit M 1 if called ﬁrst time then
P sel ← goal variables in Π ; 3 if S can = ∅ then
S can ← variables causally related to P sel ; 5 else m
M do then
candidate variables
regenerated in call return P sel ; 19 else if S
= then
GAMER-Style terminates;
Bin-Packing
Next-Fit Bin-packing (Moraru et al. 2019) consists of two algorithms, Next-Fit Decreasing Bin-Packing (NFD) and
Next-Fit Increasing Bin-Packing (NFI), which differ in the order of candidate variables O .
The sketch is given in Algorithm 2.
It has been shown empirically that Next-Fit BinPacking is effective for seeding (Franco et al. 2018).
In our implementation, an additional step of shufﬂing the causally related variables is added to inject randomness (line 11).
Causal Dependency Bin-Packing (CBP) (Franco et al.
, as sketched in Algorithm 3, is used in the main PDB construction phase.
Each pattern contains N randomly selected goal variables and iteratively adds causally related variables.
The PDB collection is sorted in the descending order with respect to the pattern length before being returned (line 14).
Compared to Next-Fit Bin-Packing, CBP is more random because candidate variable are not sorted and N can be adjusted.
Both bin-packing algorithms are implemented with more efﬁcient data structures in CPC0.
Evaluator
The evaluator plays a crucial role in the PDB construction process as it decides which PDB collections to be included in the set S cur .
The quality of a PDB collection is usually evaluated with respect to the reduction on the search tree size or the search time.
Since the search tree size is negatively related to the average heuristic value as conjected by Korf (1997), the heuristic value is used as the metric for evaluators such as average h (Edelkamp 2006) and random walk sampling (Haslum et al.
A stratiﬁed-samplingbased evaluator (Lelis et al. 2016; Barley, Franco, and Riddle 2014) that takes the predicted search time as the metric is employed in CPC2 (Franco, Lelis, and Barley 2018), whereas random walk sampling evaluator is used in CPC1.
As mentioned in (Franco et al. 2017), both evaluators yield similar results on symbolic PDBs.
An explanation is that since a symbolic PDB collection generally contains fewer
PDBs than an explicit one, the look up time of a symbolic PDB collection does not vary that much, resulting in a stronger correlation between the search tree size and the search time.
For faster evaluation, random walk sampling evaluator is also adopted in CPC0 with some modiﬁcations and extensions.
The random walk sampling evaluator samples each state by applying a number of uniformly chosen operators according to the heuristic value of the initial state.
After each step, if the state is known to be a dead, the random walk returns to the initial state.
In our evaluator, the sampling stops when either the 10,000 state size limit or the 30s time limit is reached, and unique states are taken from the sample.
In case the sample is empty when all sampled states are dead ends, the initial state is always included.
Then, the heuristic value of S cur is stored for each sample state.
To evaluate a new
PDB collection P sel , the heuristic value of P sel is compared with with the stored value on each sample state, and true is returned if and only if at least 25% states are improved.
The evaluation criteria can be summarized by
[Equation]
where m is the sample size and { s 1 , . .
, s m } is the set of sample states.
Compared to the hard threshold in the original evaluator, the ratio threshold is more accurate and ﬂexible for varying sample size.
After each evaluation, new dead ends are removed from sample state, and the heuristic value of each sample state is updated if P sel is included.
Resampling will be performed if and only if the heuristic value of the initial state increases by over 10% to save time.
The time overhead in maximizing over a set of PDB collections has been addressed by researchers (Holte et al.
2004; Barley, Franco, and Riddle 2014).
To reduce the size of S cur while maintaining the quality, the pruning of dominated collections is introduced.
The pruning progresses backwards from the last PDB collection and stores the maximum heuristic value of processed PDB collections for each sample state.
A PDB collection is pruned if and only if there is no state where its heuristic value exceeds that of any previous collection.
The pruning order is determined as such because earlier added PDB collections are more likely to be dominated by later ones.
This strategy is safe since it is quite rare for a useful PDB collection to underperform on every sample state.
Adaptive PDB Construction
The CPC process, as illustrated in Figure 1, starts with
NFD and then NFI.
Both of them run for 80s excluding the resampling time and start with a size limit of 10 8 that is scaled by 10 after each iteration.
Whenever a new PDB collection is added, resampling will be performed if the initial h rises by over 10%, and dominated PDB collections will be pruned.
The pruning is reasonable since the PDB collections formed in this phase are homogeneous and the time overhead for subsequent evaluations will be reduced.
The largest size limit where a PDB is added is recorded as S l for later use.
In CPC1, the perimeter PDB is constructed prior to the bin packing.
However, since the perimeter PDB usually takes too much time and memory to be useful and may disadvantage other algorithms, it is not adopted in CPC0.
After the seeding phase, the PDB construction process goes on until the time or the memory limit is reached.
At each iteration, a choice is made between CBP and GAMERStyle using the UCB1 formula, which is a learning policy that balances exploration versus exploitation in a multiarmed bandit problem and uniformly approaches the minimal expected cumulative regret (Auer, Cesa-Bianchi, and
Fischer 2002).
The score for each choice is calculated by ¯ x i + (cid:113) 2ln n n i , where ¯ x i is the average reward received by the choice i , n i is the number of times i is chosen and n is the total number of trials, and the choice with the highest score
[Figure]
is selected each time.
In our planner, n i and n are updated with the PDB construction time of the algorithm i , and the reward of i is also increased if the PDB is added to S sel .
The formula used in our planner is revised on the second term to the correct form so that it will not be biased to explored choices.
For CBP, the parameters N and S are decided using
UCB1 as well.
The choices for N range from 1 to the number of goal variables in Π and the choices for S are drawn from a set of sizes ranging from 10 9 to 10 35 , with the ones over 10 4 ∗ S l ﬁltered out to avoid PDBs too large to construct.
The maximum size in CPC1 is 10 20 .
GAMER-Style complements the Bin-Packings as it constructs a single PDB that contains all goal variables, whereas the Bin-Packings construct a PDB collection with goal variables distributed among them.
By integrating these algorithms, the adaptive
PDB construction process can generate a great variety of
PDBs and learn the most suitable algorithm to use as well as the best conﬁguration.
Before the A* search, the construction of any partial PDB will continue if there is excess time and memory and dominated PDB collections will be pruned.
Experiments
Both CPC0 and CPC1 are built upon an early 2017 fork of the Fast Downward planning system (Helmert 2006) with the h 2 -based pre-processor to remove irrelevant operators (Alc´azar and Torralba 2015).
The symbolic search is enhanced with the techniques proposed by Torralba et al.
The optimal STRIPS benchmarks from IPCare used for the evaluation, consisting of 12 domains (split ones included) and 240 planning tasks (20 tasks in each domain).
All experiments are conducted on an Intel i5-6200U 2.30GHz machine, under the IPC 2018 rule of 30-minute time limit and 8 GB memory limit.
The PDB construction is set to the time limit at 15 minutes and the memory limit at
GB for CPC1.
Since PDB construction is the most important stage in our planners and effective PDBs may signiﬁcantly reduce the search efforts, the time limit is raised to 18 minutes for CPC0.
Overall Results
The key statistics from our experiments are summarized in
Table 1, Table 2 and Figure 2.
The initial h values are taken from the tasks where they are available in both planners.
The expansions, the search time, the total time and the memory usage are taken from the tasks solved by both planners.
It can be seen from the tables and Figure 2(a) that CPC0 usually produces PDBs with higher initial h value.
This indicates that the reﬁned algorithms and evaluator can generate better heuristic.
While Table 2 shows that the number of tasks where less nodes are expanded in CPC0 is almost the same as the number of tasks where more nodes are expanded, Table 1 shows that CPC0 has slightly fewer node expansions on average.
Looking at Figure 2(b), the instances on CPC side lie further off the diagonal than those on CP0 side.
Moreover, CPC1 tends to expand more nodes for the tasks requiring a great number of expansions.
It is because the reﬁned process in CPC0 is able to construct PDBs of consistent quality over tasks of varying difﬁculty.
According to
Table 2 and Figure 2(c), the search time of CPC0 is generally less than that of CPC1, and the average search time of CPC is signiﬁcantly reduced, as per Table 1.
This may attribute to the pruning of dominated PDBs that reduces the look up time.
A combination of better heuristic and faster heuristic look up enables CPC0 to solve 12 more tasks.
Hence, our reﬁnements make the planner even more competitive in the context of IPC 2018.
On the other hand, the total time and the memory usage of CPC0 are increased on average as per Table 1.
It is because we have raised the time limit for PDB construction.
Also, the drop of perimeter PDB reduces the chance of solving a task during PDB construction, thereby increasing the total time and the memory usage for solved tasks.
[Table] [Table]
domain.
In each cell is the number of tasks where CPC0 outperforms CPC1 minus the number of tasks where CPC1 outperforms CPC0.
The search time of a task is counted if the difference is greater than 1s.
GAMER-Style Comparison
In CPC
[Table]
the variable selection method of GAMER-Style is changed from average h value to random walk sampling.
To evaluate the effect, both versions are tested on Nurikabe and
Snake, where GAMER-Style contributes the most.
In our experiments, other PDB construction algorithms are disabled, and both planners are set to the time limit at 900s and the memory limit at 4GB for PDB construction and are tested under the IPC 2018 rule
[Table]
It can be seen from Table 3 that the new version outperforms the original version in Snake.
Despite a slight increase in the average memory usage, there is a signiﬁcant improvement in the other statistics, especially a huge reduction in the average node expansions.
The difference is not that obvious in Nurikabe.
Although the new version has slightly more node expansions on average, mainly due to the task
[Table]
14, it has fewer expansions on 3 more tasks.
Also, the average initial h value of the new version is higher, and the initial h value is higher on 5 more tasks.
Thus, the new version is more likely to outperform the original version in Nurikabe.
In both domains, the average search time of the new version is signiﬁcantly reduced, due to not only better heuristic, but also the pruning method.
Since the PDBs constructed by GAMER-Style are homogeneous, it is very beneﬁcial to prune the dominated ones.
Related Work
GA-PDB (Edelkamp 2006) uses the the bin-packing algorithm and the genetic algorithm to construct PDB collections, which are evaluated by average heuristic value.
iPDB (Haslum et al.
2007) starts with a collection of PDBs each containing a goal variable and in each iteration constructs a new PDB by adding a causally related variable to an existing
PDB.
The PDBs are evaluated by random walk sampling and combined with the canonical function.
GAMER (Kissmann and Edelkamp 2011) performs symbolic A* search with the symbolic PDB constructed by GAMER-style algorithm.
It has an alternative option to perform symbolic bidirectional blind search, depending on the property of the planning task.
CPC2 (Franco, Lelis, and Barley 2018) is an early version of the Complementary planers.
The difference is that CPC does not perform Next-Fit Bin-Packing or GAMER-Style but uses CBP and Regular Bin-Packing (RBP) for PDB construction and performs random mutations to selected PDB collections.
The PDB size limits are drawn from the binomial distribution with the parameters dynamically adjusted.
A stratiﬁed-sampling-based evaluator is used, which evaluates the PDB with respect to the predicted search time and adds a PDB collection if the search time is predicted to be reduced.
The pruning of dominated PDB collections is performed after every certain time period.
Conclusions
In this paper, we have presented our reﬁnements on the complementary PDB construction mechanism, including
GAMER-Style, Next-Fit Bin-Packing, Causal Dependency
Bin-Packing, as well as the random walk sampling evaluator.
We have optimised the algorithms, extended the evaluator and improved the overall process.
Our experiments show that these algorithms has been well integrated and the modiﬁed planner CPC0 has a signiﬁcantly higher coverage on IPC 2018 benchmarks than the original planner
CPC1.
Therefore, the effectiveness of the CPC mechanism in cost optimal planning is again conﬁrmed and a competitive benchmark planner is contributed.
For future research, some other PDB construction algorithms and cost-partitioning strategies, e.g. saturated costpartitioning (Seipp et al. 2017), could be introduced.
Also, more advanced evaluators such as stratiﬁed sampling (Lelis et al. 2016; Barley, Franco, and Riddle 2014) may improve the PDB selection process.
A major drawback of PDBbased planners is the need of pre-processing phase, which increases the total time for solving a task.
Techniques like interleaved search and heuristic improvement (Franco and
Torralba 2019) could be further explored to address this issue.