Automatic Conﬁguration of Benchmark Sets for Classical Planning
Abstract
The benchmarks from previous International Planning Com- petitions are commonly used to evaluate new planning algorithms.
Since this set has grown organically over the years, it has several ﬂaws: it contains duplicate tasks, unsolvable tasks, trivially solvable domains, and domains with modelling errors.
Also, diverse domain sizes complicate aggregating results.
Most importantly, however, the range of task difﬁculty is very small in many domains.
We propose an automated method for creating benchmarks that solves these issues.
To ﬁnd a good scaling in difﬁculty, we automatically conﬁgure the parameters of benchmark domains.
We show that the re- sulting benchmark set improves empirical comparisons by allowing to differentiate between planners more easily.
Introduction
Domain-independent planning aims to develop general solvers that ﬁnd solutions to arbitrary sequential decisionmaking problems.
This makes the evaluation of planners an essential part of planning research.
The International Planning Competition (IPC) has set some evaluation standards and triggered the development of tools that compare planners in terms of different metrics (Linares L ´ opez, Celorrio, and Helmert 2013; Seipp et al. 2017; Vallati, Chrpa and McCluskey 2018).
The most popular metric is coverage, i e., the number of solved benchmark instances within certain time and memory limits.
Typically, there are two main goals for the evaluation: (1) analyze the impact of the novel algorithms by comparing their performance against a baseline, and (2) compare the performance against the state of the art to evaluate the progress in the area.
Evaluating planners on different benchmark sets may produce different results, leading to different conclusions from the evaluation.
Not only is it important which domains we choose, but also how we model the domains (Riddle, Holte, and Barley 2011), and which instances of a domain we select.
Therefore, having a standardized benchmark set is important to increase the comparability of results across different papers, and to avoid the use of benchmarks tailored for the proposed technique.
We focus on classical planning where the current standard benchmark set has grown across the nine editions of the IPC so far, from 1998 to 2018 (e.g., Hoffmann and Edelkamp 2005; Linares L´opez, Celorrio, and Olaya 2015).
Numerous researchers have contributed to this set by carefully designing new domains (e.g., Hoffmann et al. 2006), so it features a diverse set of domains that pose interesting challenges for planning algorithms.
However, there are several issues with this benchmark set (Moraru and Edelkamp 2019).
For example, it uses a different number of instances per domain, which reduces the value of statistics aggregated over different domains.
Moreover, instances in the current benchmark set were scaled to be useful for the evaluation of planners at the respective IPC.
Some of the domains are trivially solved by modern planners, making it impossible to show any coverage advantages over a baseline.
On the other hand, early
IPC editions did not have a specialized track for optimal planning, and some of their instances are too hard even for state-of-the-art optimal planners.
This paper deals with the question of how to generate instances of a domain to evaluate planning algorithms.
Our goal is to improve the empirical evaluation of future planning papers by (1) providing an algorithm for automatically constructing interesting benchmark sets and by (2) using this algorithm to construct a new benchmark set where differences in performance are better reﬂected in coverage than under the current standard.
We aim to generate a set of instances that range from very easy (solved by most planners) to very hard (out of reach for current state-of-the-art planners) allowing future approaches to show beneﬁts with respect to the harder instances.
This deﬁnition necessarily depends on the algorithms being evaluated.
We identify which properties are desirable for a benchmark set and propose an automatic method that generates a set of instances, given an instance generator , a baseline planner that represents the expected minimum performance of any planner, and a set of state-of-the-art planners.
The instance sets generated by our method fulﬁll the desirable properties by design.
To avoid overﬁtting to the sets of planners used and not introduce a bias in our benchmark set, our method does not select a set of instances directly, but rather performs a search on the space of possible parameters for the generator to obtain a set of instances of adequate difﬁculty.
We use our tool to design two separate sets of benchmarks, for optimal and satisﬁcing planning, and show their advantages over the current standard IPC benchmark set.
Background
Informally, a classical planning task is deﬁned by an initial state, a set of actions and a goal description.
Given a planning task, a planner ﬁnds a plan , that is, a sequence of actions that can be applied in the initial state to achieve the goal.
A plan is optimal if it minimizes the summed-up cost of the actions among all plans.
If the planner is guaranteed to ﬁnd an optimal solution, it is an optimal planner, otherwise it is a satisﬁcing planner.
In both settings, we only consider solvable planning tasks.
Since its inception in 1998, the International Planning
Competition (IPC) has set the standards for the evaluation of planners such as the planner input language PDDL (McDermott et al. 1998).
The IPC also introduced numerous planning tasks from different problem settings, called domains .
A planning task is typically divided into a domain and an instance ﬁle.
The domain ﬁle deﬁnes the types of objects, their properties, and the action schemas.
Each instance ﬁle can have a different number of objects, initial state and goals.
Most domains have an instance generator, a program that, given certain parameters and a random seed, will generate a new instance of the domain.
Even though many instance generators are available, 1 most planning papers use the benchmarks introduced for the IPCs, since a standardized benchmark set makes research more reproducible.
As an example, consider the Nomystery domain, where a truck must deliver a set of packages to certain locations.
To do that, there is a limited amount of fuel that is consumed by drive actions.
Instances differ in the amount of fuel available, the number of locations and their connections, the number of packages, and their initial and ﬁnal location.
The instance generator for Nomystery accepts several parameters that allow the benchmark designer to control the difﬁculty of the generated instances: the number of locations, the number of packages, the number of edges between locations, the maximum fuel consumption between two locations, and the constrainedness C ≥ 1 , so that the amount of fuel in the initial state is set to C times the minimum fuel consumption required to solve the instance.
Benchmark Design Principles
The purpose of a benchmark set is to evaluate planners and compare their performance on a diverse class of problems.
Ideally, one should select a diverse set of domains that are representative of real-world scenarios where different users apply planning to solve their problems.
However, in addition to selecting interesting domains, one must select a set of concrete instances from each domain to evaluate the planners on.
This selection of instances is an important step in the design of the benchmark set, since different instance sets of a domain may lead to different conclusions about which planner is better at solving instances of a given domain.
Our goal is that, for any two planners A and B (possibly unknown at the time when the instance set is generated) if A is consistently faster than B on the instances of a domain, the probability that this is reﬂected on the coverage score should be as high as possible.
IPC
New’
L
D
O
L
D
O
Nomystery
Rovers
Woodworking
Total
For aggregated statistics to be meaningful, not only should all domains have the same number of instances, but their difﬁculty should also scale similarly.
Otherwise, conclusions taken from the empirical evaluation may be biased.
Table 1 shows an example comparing 3 planners in 3 domains when using the IPC instances and our New’14 benchmark set, as described in the evaluation section.
A paper evaluating these planners with IPC instances would reach the conclusion that Decstar is clearly superior to the other two planners in these domains, both in total coverage and on a per-domain basis since it has better or equal coverage in all domains.
However, this conclusion is biased because instances are not well scaled.
Instances in Rovers and Woodworking are way too easy and therefore they do not show any differences between the planners.
Using our New’ instances leads to a different conclusion: all three planners are complementary.
Of course, no strong conclusions can be taken out of only 3 domains.
However, using more domains will help to alleviate this issue only if the instances are well scaled.
A good scaling must meet three conditions: (1) have easy instances that are solved by all planners, (2) have hard instances that are not solved by any current planner, and (3) the instance difﬁculty should grow smoothly.
Condition (1) is necessary for experiments to be informative at all: if some planners do not solve any instance, no conclusions can be obtained about their relative performance.
This happens in some domains of the IPC benchmark set for optimal planning.
E.g., Fi ˇ ser, Torralba, and Shleyfman (2019) write that “In childsnack, [they] measured about twice as many expanded states per second.
However, no planner solved any instance in this domain.”.
Condition (2) is necessary for new algorithms to show that they can deal with instances that previous planners could not, as shown by our example in Table 1.
Condition (3) is necessary for differences between the planners’ performance to be reﬂected in coverage.
To see why, consider an idealized setting where a baseline planner
A , whose runtime scales exponentially ( t ( A, x ) = x C for some constant C ), is compared to an improved planner version B which is always faster than A by at least a factor of
K > 1 , i.e., t ( B, x ) ≤ t ( A,x ) K .
Given these assumptions, there is a guaranteed difference in coverage if and only if (1) some instances are solved by B , (2) not all instances are solved by A , and (3) K ≥ C .
Otherwise, there may be cases where both planners solve the same number of instances,
[Figure] [Figure]
Complementary 2, IPC
Delﬁ-blind, IPC
Complementary 2, New’
Delﬁ-blind, New’ and the difference in performance by a factor of K is missed by the coverage analysis.
If these conditions do not hold, it is possible to choose runtimes for A and B that are compatible with these exponential scalings, so that their coverage is equal.
For example, if K = 2 and C = 3 , then (3) does not hold.
So for any time limit (e.g., 300 seconds), if the runtime of the last instance solved by A is close enough to the time limit (e.g., 250 seconds), the next instance cannot be solved by B below the time limit (e.g., 250 · 3> 300 ).
Real distributions of planner runtimes over sets of instances differ from this idealized example in that they usually involve constant factors, the runtime scaling of different planners may be completely different, and even for a single planner it may be impossible to obtain instances that scale according to the desired runtimes in some domains.
But ideally, all domains should consist of a collection of instances of increasing difﬁculty, ranging from very easy to very hard for current planners.
Therefore, we aim for a collection where the easiest instance is quickly solved by most planners; all domains have instances that are not solved by current planners; and difﬁculty scales by approximately a factor of 1.5–2 between consecutive instances.
Figure 1 exempliﬁes why a smooth scaling is important in practice.
The plot shows the runtimes of two optimal planners on the Barman domain from IPC 2011 and our New’ benchmark set.
In the IPC instances the difﬁculty does not grow smoothly.
Instead, for each group of four instances the difﬁculty increases visibly and the runtime of all planners increases by about one order of magnitude.
This is undesirable since we cannot observe differences in performance for some planners by inspecting their coverage.
In contrast, the difﬁculty on the new benchmark set grows more smoothly, there are instances of more varied difﬁculty for all planners, and fewer jumps in their runtime.
Accordingly, now we can observe that Complementary 2 is able to solve some instances that are not solved with Delﬁ-blind in this domain.
Given our deﬁnition of an ideal benchmark set as one that meets conditions (1), (2), and (3) described above, the instance selection necessarily depends on the planning algorithms being evaluated.
As our objective is to generate a benchmark set to evaluate future planners that do not exist yet, one cannot directly select instances that are useful to compare planner’s right now.
However, selecting an instance set that scales well for current planners may generalize well for planners that are introduced in the next few years.
Indeed, our New’14 instance set featured in our examples of
Table 1 and Figure 1 was conﬁgured without using any planner after 2014, so it did not use any information regarding the IPC’18 planners mentioned in our examples.
One must be careful not to “overﬁt” the benchmark set to match the set of selected planners, so that the difﬁculty scales well for the selected planners but not for future planners.
To avoid overﬁtting, we impose two restrictions on the benchmark conﬁguration process.
On the one hand, the optimization process does not consider concrete instances, but rather only decides which overall characteristics they should have (e.g. the number of objects in each instance).
The ﬁnal benchmark set is then generated with a random seed different from the one used during our optimization process.
On the other hand, we do not consider the individual results of all planners available for the optimization.
In each domain, we require a baseline planner that represents the expected minimum performance of any planner to ensure that some instances are solved by all planners.
Also, to ensure that some instances remain unsolved, we estimate the performance of state-of-the-art planners by taking the minimum runtime of any of the available planners on each instance.
This makes our instance selection as objective as possible since it does not depend on the concrete set of planners available to the benchmark designer, as long as the best planner for the given domain is considered.
Conﬁguration of Planning Benchmarks
We consider domains that have an instance generator with several parameters to control the hardness of the generated instances.
4. Framework
We model the problem of generating the instances of a benchmark set as follows.
Our tool takes as input a tuple ( spec , G, A , B ) , where spec is a domain speciﬁcation; G is an instance generator; A is a set of state-of-the-art planners; and B is a set of baseline planners.
The output will be a set of instances of that domain.
The domain speciﬁcation describes the instance generator parameters and their constraints and it is discussed in detail in the next section.
The instance generator G is a function that takes as input a tuple of parameter values ρ = (cid:104) ρ 1 , . .
, ρ k (cid:105) and a random seed seed ∈ N + and outputs a planning task.
Let p ∈
A ∪ B be a planner, and Π a planning task produced by G ( ρ, seed ) for some seed ∈ N + .
We deﬁne t ( p, Π) as the runtime of planner p to solve task Π .
For every instance we characterize the performance of a set of planners
A on an instance Π as the minimum runtime of any planner, t ( A , Π) =
min p ∈A t ( p, Π) .
Given a parameter conﬁguration ρ , we deﬁne t ( p, ρ ) as the average runtime over all possible tasks that the generator may output for different random seeds.
We estimate t ( p, ρ ) by sampling k tasks Π 1 , . . .
, Π k from the distribution and taking the average running time (cid:80) i ∈ [1 ,k ] t ( p, Π i ) k .
In practice, a small k is sufﬁcient for most domains.
In our experiments we used k = 1 , which was enough to produce robust results.
Our goal is to select a set of parameter conﬁgurations ρ 1 , . .
. , ρ n such that the running time t ( A , ρ i ) scales smoothly with i , as described in our general design principles.
Note that our automatic tool is not allowed to hand-pick the random seed, but rather the ﬁnal benchmark set is created by sampling these distributions of instances with new random seeds.
This helps to avoid overﬁtting.
An assumption is that the variance of the runtimes t ( A , Π i ) for tasks generated with the same parameter conﬁguration is not too high, since otherwise the parameters provided to the generator are irrelevant for obtaining a smooth difﬁculty scaling.
This is a reasonable assumption in practice, following analyses made in the context of predicting planner runtimes (de la
Rosa, Cenamor, and Fern´andez 2017).
Out of 10 domains analyzed by de la Rosa, Cenamor, and Fern´andez, most of them had a very low variance.
The one with highest variance was Barman, where 90% of the instances were still relatively close to the average runtime, especially for our purposes.
4. Domain Speciﬁcation
In order to use our benchmark conﬁguration tool, the benchmark designer must specify how to call the instance generator, what parameters are available, as well as which values are appropiate for each parameter.
We distinguish between two types of parameters.
Linear parameters can be assigned arbitrary non-negative numeric values, where larger values usually result in harder instances.
They are typically used to specify the number of objects of a given type.
Each generator should have at least one linear parameter that helps to control the difﬁculty of the generated instances.
In contrast, enumerated parameters have a ﬁnite set of values, and we do not make any assumption about their impact on instance hardness.
All other parameters are ﬁxed to a predeﬁned constant value.
We deﬁne the instances of a domain as a set of sequences of instances.
A sequence consists of a list of planning tasks Π 1 , Π 2 , . . .
of increasing difﬁculty.
To ensure that difﬁculty increases, all instances in the sequence have a ﬁxed value for all enumerated parameters, whereas the value of linear parameters increases linearly across the sequence.
We specify this via the base value b that the linear parameter takes for Π 1 and the slope m .
For example, suppose that a domain has two linear parameters that deﬁne the number of packages ( b = 2 , m = 1 ), and trucks ( b = 1 , m = 0 . 5 ).
Then, the sequence will generate instances with the following numbers of packages and trucks: (2 , 1) , (3 , 1) , (4 , 2) , (5 , 2) , (6 , 3) , etc.
Considering sequences of instances allows us to choose the parameters that generate instances which current planners fail to solve within reasonable time.
A limitation of this approach is that not all combinations of parameters are possible.
In the example above, a single sequence cannot contain both (3 , 2) and (2 , 3) because that would require to decrease one of the parameters, which is not allowed by our linear scaling.
In most cases, this is not a problem because we can use multiple sequences of instances for a single domain.
A notable exception are parameters that deﬁne the
[Table]
[Figure]
width and height of a grid, because they have a strong interaction, i.e., the number of cells is the product of both parameters.
In that case, we consider them a single parameter so that the number of tiles in the grid scales linearly.
The snippet from Figure 2 shows the domain speciﬁcation for the nomystery domain.
For each linear parameter, lower and upper bounds for the base and slope values should be provided.
This allows the domain modeller to specify preferences on which parameters to scale (e.g., by restricting the slope m for the number of locations to be between 0.1 and 1, they indicate their preference to increase difﬁculty by scaling the number of packages).
Note that this is important, since a property of a good benchmark set is that instances reﬂect problems that are “interesting in practice”, and this is a subjective matter that the conﬁguration tool cannot decide on its own.
If the benchmark designer has no such preference, all parameters can be left with a default interval.
Often, instance generators impose constraints on the range of parameter values or their combination.
Those constraints must be enforced by adding a postprocessing function that updates the value of the parameters passed to the generator.
This is an arbitrary function provided by the benchmark designer which receives the parameters that were automatically chosen and outputs the ﬁnal parameters that will be provided to the generator.
For example, if the number of packages has to be greater than the number of locations, instead of directly selecting the number of packages, our linear scaling will consider the number of locations and the number of additional packages.
All of these adjustments must be done on a per-domain basis, since they depend on the speciﬁc characteristics of the domain and generator.
Given this framework, our automatic tool decides which sequences of instances are suitable for each domain.
This is done in two phases: the ﬁrst phase designs a set of candidate sequences (Sequence Optimization), and the second phase performs a ﬁnal selection that adheres to our design principles as much as possible (Sequence Selection).
4. Sequence Optimization
The ﬁrst phase generates sequences of 30 instances by optimizing sequence parameters.
To guide the search towards sequences that scale the instance difﬁculty as smoothly as possible, we compute a penalty score for each sequence and search for the sequence that minimizes this score.
Sequences are evaluated by running hand-picked state-ofthe-art ( A ) and baseline ( B ) planners on their instances, using a time limit of 180 seconds per instance.
We ignore instances that are solved under 10 seconds, considering that differences of ± 5 seconds are not meaningful enough.
Since the sequences are generated with increasing values of the linear parameters, we assume that the runtimes will always increase, so we can stop our evaluation as soon as one instance is not solved under the time limit.
In cases where this does not hold, we enforce it by sorting the runtimes of the instances.
Our assumption is that these anomalies stem from using different random seeds for the instance generator and the results will be different with different random seeds.
The runtime of a set of planners is the minimum of the runtimes of the individual planners.
For evaluating a sequence we consider the ﬁrst ﬁve instances with a minimum runtime above 10 seconds.
We ignore harder instances because they will usually incur runtimes above the 180 seconds time limit.
Let t ( X, 1) , . . .
, t ( X, 5) be the runtimes of the set of planners X on the ﬁrst ﬁve instances with a runtime above 10 seconds.
The penalty score is deﬁned as (cid:80) i ∈ [2 , 5] S ( B , i ) + S ( A , i )
where S ( X, i ) =   3 − 2 t
( X, i ) t ( X, i − 1) if 1 ≤ t ( X, i ) t ( X, i − 1) ≤ 1 .
if 1 . 5 < t ( X, i ) t ( X, i − 1) ≤1 − 2 t ( X, i − 1) t
( X, i ) if 2 t ( X, i − 1) ≤ t ( X, i ) ≤ if t ( X, i ) >
This penalty is lower for sequences whose runtime scales smoothly, assigning a minimum score of 0 to any sequence where the runtimes of both the baseline and state-of-the-art planners scale exponentially with a factor between 1.5 and 2, e.g., (cid:104) 10 , 15 , 23 , 35 , 52 , . . .
(cid:105) , or (cid:104) 10 , 20 , 40 , 80 , 160 , . . .
(cid:105) .
If not enough instances are solved in the [10 , 180] second interval, the sequence gets a penalty of 2, and otherwise we assign it a penalty between 0 and 1.
To avoid generating sequences where all instances are solved by the state-of-theart planners, we also add a penalty of 1 for each instance solved by them beyond 20 instances.
To guarantee that all valid sequences contain some instances solvable within the time limit and to speed up the evaluation we require the ﬁrst three instances to be solved within 10, 60, and 180 seconds, respectively.
Otherwise, we discard the sequence, unless all linear parameters are at their minimum value.
The concrete choice of penalty values is arbitrary.
What matters is that sequences that minimize this score adhere more to the design principles introduced in Section 3 than those that do not, thereby guiding the parameter optimization towards good sequences.
4. Sequence Selection
After performing one or more optimization runs for a domain (using different random seeds) as described above, we collect all sequences seen during the optimization process.
Since this set can be very large, we only keep the 100 sequences with the lowest penalty score per value of the enumerated parameters.
For each group of sequences where the planners solves the same instances, we only keep one member of the group.
This ﬁltering ensures that we keep a set of diverse sequences with a good penalty score.
For each sequence, we collect the runtimes of all instances solvable in 180 seconds from the sequence optimization phase.
For the rest of the instances, we estimate the runtime by assuming that runtimes will increase according to the average increasing factor t ( A , i ) /t
( A , i − 1) observed on the instances solved between 5 and 180 seconds.
This is a very rough estimate but it is accurate enough for the purposes of choosing up to when a sequence should be continued (see below).
Wemodel the problem of selecting a suitable set of subsequences as a Mixed-Integer Programming (MIP) problem, where constraints directly aim to model the design principles of Section 3.
The decision variables model the start and end points of each sub-sequence of instances.
The selection must satisfy the following hard constraints that model properties desirable for a good set of instances:
(H1)
The number of selected instances must be exactly 30.
(H2)
There must be at least one instance solvable by the baseline under 30 seconds.
(H3)
All sequences must start with an instance that is solvable by a state-of-the-art planner and end with an instance whose estimated runtime is higher thanseconds.
(H4)
Each parameter conﬁguration must be used (with different random seeds) at most twice, and only once for domains whose generators do not admit a random seed.
The objective is to minimize the summed-up penalty score of all sequences used, plus the penalty incurred for violating any of the following soft constraints :
(S1)
The number of instances solved by the baseline under 30 seconds must be between 2 and 6 (with a penalty of 2 x 2 where x is the deviation with respect to the constraint).
(S2)
The number of instances solved under 180 seconds must be between 8 and 15 (with a penalty of 2 x 2 where x is the deviation with respect to the constraint).
(S3)
All sequences must end with an instance whose estimated runtime is between 18 000 and 180 000 seconds (that is, 1–2 orders of magnitude larger than the typical time limit of 30 minutes).
Larger times t incur a penalty of 100 t/ 180000 and smaller times incur a penalty of 100(18000 /t ) .
(S4)
If a parameter conﬁguration is used more than once, there is a penalty of 100.
[Table]
For domains where all instances in the sequence are solved by state of the art planners under 180 seconds (because the domain is solvable in polynomial time and it is impossible to fulﬁll our criteria with state-of-the-art planners), we consider the runtimes of the baseline instead of those of the state of the art planners in our constraints described above
[Table]
Constraints (H2), (S1) and (S2) ensure that the instance set contains some easy instances, so that any future planning algorithms are expected to solve at least some instances, allowing researchers to analyze the behaviour of their algorithms in the domain.
Constraints (H3) and (S3) ensure that, whenever possible, at least some of the instances are expected to be out of reach for state-of-the-art planners.
Together with minimizing the penalty score of the selected sequences, they aim to obtain a smooth scaling, since sequences must interpolate between easy and hard instances and sequences with smoother scaling are preferred.
Finally, constraints (H4) and (S4) are needed to avoid duplicate instances and instances that are very similar to each other.
The penalties are set arbitrarily, but they scale quadratically with respect to the deviation because it is better to not fulﬁll several soft constraints entirely than to completely ignore one of the constraints.
Experiments
We implemented the ﬁrst phase, i.e., sequence optimization, using the automatic conﬁgurator SMAC (Hutter, Hoos, and
Leyton-Brown 2011).
We test our approach by running two completely separated optimizations for optimal and satisiﬁcing planners.
As baseline planners, we use blind search for optimal planning and greedy best-ﬁrst search with the FF heuristic (Hoffmann and Nebel 2001) for satisﬁcing planning, both implemented in Fast Downward (Helmert 2006a).
Both for optimal and satisﬁcing planning, we generate two separate benchmark sets, New’14 and New’20, that differ in the set of state-of-the-art planners available for optimization.
The New’14 set consists of a heterogeneous set of planning algorithms from IPCs 2011 and 2014.
The New’ set is trained using the same planners plus the ones used in the evaluation.
Therefore, for New’14, the conﬁguration and evaluation sets are disjoint, while for New’20, the sets overlap.
Table 2 gives an overview of the planners used.
Since we limit each planner run during the optimization to 3 minutes, we adapt the planners for the conﬁguration phase by breaking portfolios into components and by adapting preprocessing time limits.
For optimization in each domain, we hand-pick 1–3 planners that perform best in that domain, which is sufﬁcient to approximate the minimum time of any planner in each instance.
We run SMAC 10 times using different random seeds.
Each run is limited to 10 hours.
After the ﬁrst phase ﬁnishes, we consider all sequences encountered during optimization for the second phase, i.e., sequence selection.
We ﬁlter the instances as described in
Section 4.4 and solve the MIP for sequence selection using
CPLEX 12.10, which ﬁnishes in under 30 seconds for each domain.
We evaluate the new benchmark sets using the aforementioned planners, limiting each run to 30 minutes and 3.
GiB. In Table 3 we compare the IPC benchmarks to the new benchmark sets for optimal and satisﬁcing planning.
We compare benchmark sets according to two metrics: the range of coverage scores per domain, which allows us to see how many instances are solved by all planners and how many remain unsolved by any of the planners; and the number of pairwise comparisons in which a planner had higher coverage than another, which quantiﬁes how many differences in the performance of planners are reﬂected by the coverage score.
In optimal planning the difference between the benchmark sets is rather subtle because difﬁculty typically scales very fast with increasing instance size.
Therefore, the IPC set has some interesting instances in all domains.
Also, it can be very hard to generate instance sequences whose difﬁculty scales smoothly, since often increasing one of the parameters of a generator by a unit has a big impact on runtime.
The results are more pronounced for satisﬁcing planners, where the IPC set scales very poorly for some domains.
Only in the Elevators domain the IPC set is superior in terms of comparisons detected by the coverage score compared to both new benchmark sets.
In contrast, with the new benchmark sets, we observe differences in performance in domains like Blocksworld, Driverlog or Zenotravel, where all planners solve all instances in the IPC set.
Overall, New’ uncovers more differences in coverage between pairs of planners than the IPC set in 21 out of 26 domains, while
[Table]
the opposite is the case in only 4 domains.
The comparison between New’14 and New’20 reveals that our technique is not very sensitive to the set of stateof-the-art planners.
The reason is that the state of the art has not advanced enough in the last six years to make a set of instances trained with our method in 2014 outdated.
Discussion
Our paper deals with the problem of generating instances that are adequate to evaluate planning algorithms.
The goal is to select instances that scale well, so that differences in algorithm performance are reﬂected in the number of problems solved within a certain time limit.
It must be remarked that no benchmark set can replace a careful analysis of the results.
Aggregating results from different domains without further analysis may be misleading and an empirical analysis only based on total coverage should be discouraged.
Nevertheless, coverage is a useful metric to summarize experimental data and it is used by most planning papers.
As shown by our experiments, the coverage metric is more meaningful for the benchmark sets generated with our approach than with the previous standard.
Our main result is a new benchmark set, as well as a set of generators and tools that can be used in the future to automatically generate new instances.
In other communities like SAT, there has been a lot of research on how to construct random instances (Selman, Mitchell, and Levesque 1996; Achlioptas et al. 2000;
Gir´aldez-Cru and Levy 2015; Xu et al. 2005) around the phase transition (Cheeseman, Kanefsky, and Taylor 1991).
Our approach is orthogonal to any approach that can generate new instances, e.g., around the phase transition of planning problems (Rintanen 2004; Rieffel et al. 2014), or with suitable initial states and goals for Sokoban (Bento, Pereira, and Lelis 2019).
Those approaches provide an instance generator that adjusts the instance difﬁculty for a given problem size, but to generate an instance set still requires to select the value of certain parameters.
Our approach is complementary, since it can be used to select suitable values that are useful to evaluate a given set of solvers.
Our tool can also be adapted to generate benchmarks with different characteristics, e.g., with smaller instances that are solved in a few seconds (Ruml 2010).
Future work could also consider relations among different domains theoretically (Helmert 2003, 2006b) or empirically (Cenamor and Pozanco 2019).
Acknowledgments
We thank Florian Pommerening for helping us set up the experiments and we thank the anonymous reviewers for their helpful comments.
We have received funding for this work from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 817639).
´Alvaro Torralba was employed by Saarland University and the CISPA Helmholtz
Center for Information Security during part of the development of this paper.