Original: On the State-of-the-Art in Cost-Optimal Planning: Symbolic Planning Pattern Databases vs. Portfolio Planning
Revised: Cost-Optimal Planning in IPC 2018: Symbolic Planning Pattern Databases vs. Portfolio Planning

Original: Abstract
Revised: Abstract

Original: The optimal trackcertainly is  the most attracted and probably the most prestigious event in the International Planning Competition (IPC).
Revised: The optimal track is one the most exiting events in the Inter- national Planning Competition (IPC).

Original: In this position paper we stress that  symbolic search and pattern databases were  the dominating planning approaches in the latest IPC in 2018, and in continuation to the precursor IPC in 2014 should be considered as  the current state-of-the-art.
Revised: In this position paper we argue that —despite of not winning the competition— symbolic search and pattern databases were likely the most inﬂuential planning approaches in the latest IPC in 2018, and, in continuation to the precursor IPC in 2014, should be considered as candidates for the current state-of-the-art.

Original: Five of the Top 6 planners in the 2018 com- petition, namely Complementary (1 and 2), Planning-PDBs, Symbolic-Bidirectional, and Scorpion areall based on these technologies.
Revised: Five of the Top 6 planners in the 2018 competition, namely Complementary (1 and 2), Planning-PDBs, Symbolic-Bidirectional, and Scorpion are based on these technologies.

Original: These planners use the same technology across all domains and plan in one state space.
Revised: These planners use the same technology across all domains and plan in one state space.

Original: The winner of IPC 2018 with an ≈ 1% lead in problems being solved, however, is a so-called portfolio planner, consisting of a selection of 17 different planners, one of which is chosen in a classiﬁer that was trained on a manually selected set of benchmark instances.
Revised: The winner of IPC 2018 with an ≈ 1% lead in problems being solved, however, is a so-called portfolio planner, consisting of a selection of many different planners, one of which is chosen in a classiﬁer that was trained on a manually selected set of benchmark instances.

Original: In about half of its successful runs, it called the winner of the previous IPC.
Revised: In about half of its successful runs, it called the winner of the previous IPC, which in turn is based on symbolic search.

Revised: We argue on whether or not to exclude portfolios from the IPC is possible and wanted.

Original: Introduction
Revised: Introduction

Original: Being inspired by the ultimate goal of general problem solving, in the ﬁeld of action planning, there is a common understanding that planning competitions ran on a set of unknown and partly new set of benchmark problems, advance planning technology the most.
Revised: Being inspired by the ultimate goal of general problem solving, in the ﬁeld of action planning, there is a common understanding that planning competitions ran on a set of unknown and partly new set of benchmark problems, advance planning technology the most.

Original: Starting in 1998, over the years considerable progress has been made in the development of new planners, due to competitors being brave enough to face a coding contest with an unknown benchmark set and with the obligation to offer their source code.
Revised: Starting in 1998, over the years considerable progress has been made in the development of new planners, due to competitors being brave enough to face a coding contest with an unknown benchmark set and with the obligation to offer their source code.

Original: Over the years, they have been confronted with an increasing set of challenges, both in the planning domain description language (PDDL) expressiveness, in problem metrics, intheir inherent complexity and in scaling problem sizes.
Revised: Competitors have been confronted with an increasing set of challenges including extended expressiveness of planning domain description language (PDDL), involved planning task metrics, in inherent problem complexity, and in scaling problem sizes.

Original: While several tracks were spawned,the probably most attractive one is the deterministic part of the IPC, and its track on cost-optimal planning.
Revised: While several tracks were spawned, one is the deterministic part of the IPC, and its track on cost-optimal planning.

Original: The outcome of the optimal track in the most recent
Revised: The outcome of the optimal track in the most recent

Original: International Planning is revisited in Table 1 1 .
Revised: International Planning is revisited in Table 1 1 .

Original: The corresponding output on IPC 2014 has been provided and discussed by (Edelkamp, Kissmann, and Torralba 2015).
Revised: The corre- sponding output on IPC 2014 has been provided and discussed by (Edelkamp, Kissmann, and Torralba 2015).

Original: While most planners present one sole planning technology, the winning planner Delﬁ (Sieverset al. 2019) is a Portfolio , a mixture of different technologies.
Revised: While most planners present one sole planning technology, the winning planner Delﬁ (Sievers 2019) is a portfolio , a mixture of different technologies.

Original: Given a problem task, it selects a planner based on a classiﬁer, being trained on a manually chosen set of known planning benchmark instances.
Revised: Given a problem task, it selects a planner based on a classiﬁer, being trained on a manually chosen set of known planning benchmark instances.

Original: In its set of planners, 16 were contained as part of the Fast Downward planning framework, including at least one (Canonical PDB) that used pattern databases.
Revised: In its set of planners, 16 were contained as part of the Fast Downward planning framework, including at least one (Canonical PDB) that used pattern databases.

Original: The most effective approach chosen by this classiﬁer, however, was the symbolic search planning system, which won the precursor 2014 IPC competition.
Revised: The most effective approach chosen by this classiﬁer, however, was the symbolic search planning system, which won the precursor 2014 IPC competition.

Original: In the only domain, where Delﬁ scored overall best (see results for Petri Net Alignment), it called this planner.
Revised: Moreover, in the only domain, where Delﬁ scored overall clear best, it called this planner.

Original: From the individual planner algorithmic side, in portfolios there is often little that is novel, the contribution of these planners is often found in the machine learning algorithm, which is trained on a set of known planning problems and that eventually selects the planner conﬁguration to call. 
Revised: From the mere planning side, in portfolios there is often little that is novel, the contribution of these planners is often found in the machine learning algorithm, which is trained on a set of known planning problems and which eventually selects the planner conﬁguration to call. In Delﬁ one planner is called per instance.

Original: In terms of the potential of different planning approaches available to Delﬁ, the IPC outcome with a lead ofonly two more being solved ( ≈ 1% of 200 benchmark problems) is quite small, showing that cost-optimal planning is tough, even for portfolios.
Revised: In terms of the potential of different planning approaches available to Delﬁ, the IPC outcome with a lead of two more being solved ( ≈ 1% of 200 benchmark problems) is quite small, showing that cost-optimal planning is tough, even for portfolios.

Original: A change in one domain would have resulted a different outcome.
Revised: A change in one domain would have resulted a different outcome.

Original: While portfolio planning is in alignment with the rules of the competition, one underlying issueto be discussed in this paper is that the teams of other participating planners opted against using portfolio technology for one sole planning approach to avoid blurring the scientiﬁc outcome in favour of  a clearer picture on what technology is currently leading.
Revised: While portfolio planning was in alignment with the rules of the competition, one underlying issue is some participating planners avoided using portfolio technology, presumably in favour of getting a clearer picture on what technology is currently leading.

Original: They  prefer working on new technologies and refused going in for a mixture of existing planners.
Revised: They seemed to prefer working on new plan search technologies, instead of going in for a mixture of existing planners.

Original: Given the outcome of the competition and the different type of contributions available in portfolio and non-portfolio planners, people interested in planning especially outside to the core  community have to be warned not to derive wrong scientiﬁc conclusions by simply looking at the outcome.
Revised: Facing the outcome of the competition and the different type of contributions available in portfolio and non-portfolio planners, people interested in planning especially outside to the core planning community have to be warned not to derive wrong scientiﬁc conclusions by only looking at the outcome.

Original: Competition results always have to be dealt with care.
Revised: Competition results always have to be dealt with care.

Original: The stress of this paper, therefore, is to argue that that the two technologies of BDD-based symbolic search planning (Edelkamp and Helmert 2001) and planning pattern databases (Edelkamp 2001)ﬁrst joint up by (Edelkamp 2005) dominated the competition, and have demonstrated state-of-the-art in 2018.

Original: This position paper a) provides a clearer picture on what is the currently leading technology according to IPC 2018 andb) discusses on whether or not a portfolios help to push or blur the outcome of a competition. 
Revised: This position paper aims to provide a clearer picture on what is the currently leading technology according to IPC 2018 and discusses on whether or not a portfolios help to push or blur the outcome of a competition. The stress of this

Original: AgriCal
Revised: AgriCal

Original: Data Net
Revised: Data Net

Original: NuriOrganic
Revised: NuriOrganic

Original: Petri Net
Revised: Petri Net

Original: Settcola dera
Revised: Settcola dera

Original: Network kabe
Revised: Network kabe

Original: Synthesis
Revised: Synthesis

Original: Alignment lers
Revised: Alignment lers

Original: Snake
Revised: Snake

Original: Spider
Revised: Spider

Original: Termes Σ
Revised: Termes Σ

Original: Delﬁ
Revised: Delﬁ

Original: Complementary Complementary
Revised: Complementary Complementary

Original: Planning-PDBs
Revised: Planning-PDBs

Original: SymbBiDir
Revised: SymbBiDir

Original: Scorpion
Revised: Scorpion

Original: Delﬁ
Revised: Delﬁ

Original: FDMS FDMS
Revised: FDMS FDMS

Original: DecStar
Revised: DecStar

Original: Metis
Revised: Metis

Original: MSP
Revised: MSP

Original: Metis
Revised: Metis

Original: ExplBlind
Revised: ExplBlind

Original: SympleSymple-            maplanmaplan-            
Revised: SympleSymple-            maplanmaplan-            paper, therefore, is to argue that, while not winning the competition because of portfolio planning, the two technologies of BDD-based symbolic search planning (Edelkamp and Helmert 2001) and planning pattern databases (Edelkamp 2001) ﬁrst joint up by (Edelkamp 2005) seemingly dominated the overall outcome competition.

Original: Symbolic Search and Pattern Databases
Revised: Symbolic Search and Pattern Databases

Original: The IPC 2018 planner Symbolic-Bidirectional (SymbBiDir), suggested  as a baseline planning technology, includes no lower bound at all, it relies on so-called blind search, i.e., a search with no heuristic search guidance.
Revised: The IPC 2018 planner Symbolic-Bidirectional (SymbBiDir) was suggested by the authors and accepted by the organizers as a baseline planning technology. It includes no lower bound at all and, thus, relies on so-called blind search, i.e., a search with no heuristic search guidance.

Original: As actions carry cost, instead of a breadth-ﬁrst exploration this induces a cost-ﬁrst traversal of the state-space graph.
Revised: As actions carry cost, instead of a breadth-ﬁrst exploration this induces a cost-ﬁrst traversal of the state-space graph.

Original: In symbolic planning , the core difference to explicitstate space planning is the use of binary decision diagrams (BDDs) to represent state sets in the search compactly (Bryant 1986).
Revised: In symbolic planning , the core difference to explicitstate space planning is the use of binary decision diagrams (BDDs) to represent state sets in the search compactly (Bryant 1986).

Original: As actions can also be represented in form of BDDs encoding the transition relation, it is possible to progress and regress  in this succinct functional state set representation to perform forward and backward exploration in an operation called relational product (Clarke et al. 1996).
Revised: As actions can also be represented in form of BDDs encoding the transition relation, it is possible to progress and regress planning state in this succinct functional state set representation to perform forward and backward exploration in an operation called relational product (Clarke et al. 1996).

Original: An A*-type algorithms has been proposed by (Edelkamp and Reffel 1998).
Revised: A ﬁrst A*-type algorithm for BDDbased heuristic search has been proposed by (Edelkamp and Reffel 1998).

Original: Memory savings  in turn often leads to  savings in CPU time.
Revised: One obsevation is that memory savings obtained via the compact representation in a BDD in turn often also lead to signiﬁcant savings in CPU time.

Original: The gain of a symbolic representation in IPC 2018 is ampliﬁed, when comparing the performance gap of SymBiDir with ExplicitBlind search.
Revised: The gain of a symbolic representation in IPC 2018 is ampliﬁed, when comparing the performance gap of SymBiDir with the other baseline planner ExplicitBlind.

Original: Indicated by the name, the twoblind search baseline planners are not executing theexact same exploration,mainly due to the fact that coding regression search is notat all immediate for the usual partial goal state representation given; so that the latter conducts a forward state-space traversal only.
Revised: As the names indicate, the two baseline planners are not executing the same exploration, due to the fact that coding regression search is not immediate for the usually given partial goal representation; so that the latter conducts a forward state-space traversal only.

Original: To the contrary, SymBiDir executes bidirectional costbased search, much in the sense of bidirectional application of Dijkstra’s single-source shortest path algorithms (Dijkstra 1959), taking care of the fact that the optimal solution might not be established on the ﬁrst meeting of both search frontiers.
Revised: To the contrary, SymBiDir executes bidirectional costbased search, much in the sense of bidirectional application of Dijkstra’s single-source shortest path algorithms (Dijkstra 1959), taking care of the fact that the optimal solution might not be established on the ﬁrst meeting of both search frontiers.

Original: As the BDDs represent state sets, recursive solution construction is needed for extracting optimal plans.
Revised: As the BDDs represent state sets, recursive solution construction is needed for extracting optimal plans.

Original: Aspects like a partitioned computation of successors (called the image ), variable ordering based, as well as the inclusion invariant constraints to rule out illegal and dead-end states turn out to be crucial factors to improve the exploration efﬁciency (Torralba et al. 2017).
Revised: Aspects like a partitioned computation of successors (called the image ), variable ordering based, as well as the inclusion invariant constraints to rule out illegal and dead-end states turn out to be crucial factors to improve the exploration efﬁciency (Torralba et al. 2017).

Original: The performance results of SymBiDir revealed, that in only two of the ten domains (Snake, Spider) is was not doing well, otherwise this suggested baseline planner would have won the competition!
Revised: The performance results of SymBiDir revealed, that in only two of the ten domains (Snake, Spider) is was not doing well, otherwise the baseline planner would have won the competition!

Original: This indicates the power of symbolic state space representation, and suggests that  the vast amount of reﬁned heuristics for planning does not necessarily lead to a throughout leading technology.
Revised: This indicates the power of symbolic state space representation and exploration, and suggests that at least across the entire IPC 2018 benchmark set, the vast amount of reﬁned heuristics for planning do not always lead to a throughout leading technology.

Original: According to the result of the 2018 competition, planning pattern databases (PDBs) (Edelkamp 2001) are one of the few exceptions.
Revised: According to the result of the 2018 competition, planning pattern databases (PDBs) (Edelkamp 2001) appears to be one of the few exceptions.

Original: The combination of PDBs and symbolic search in the planners Complementary (1/2) and PlanningPDB aresigniﬁcantly outperformingSymbolic Bidirectional
Revised: On the testbet of IPC 2018 the combination of PDBs and symbolic search in the planners Complementary (1/2) and Planning-PDB are outperforming

Original:  Search.
Revised: Symbolic Bidirectional Search.

Original: The former two are inspired by results of (Franco et al. 2017), while the latter improves on bin packing algorithms for the pattern selection problem.
Revised: The former two are inspired by results of (Franco et al. 2017), while the latter improves on bin packing algorithms for the pattern selection problem.

Original: Besides a major rewrite, one new feature  is that the forward search is explicitstate, while  the backward traversal is symbolic.
Revised: Besides a major rewrite, one new feature of these new planners is that the forward search is in fact explicit-state, while only the backward traversal is symbolic.

Revised: Of course, many heuristics besides PDBs are still worth further investigation.

Revised: For the case if Snake and Spider were removed, it is difﬁcult to draw a conclusion on different types of heuristics from the IPC-18 results.

Revised: There is no free lunch.

Revised: But the overall performance of bidirectional symbolic search is surprisingly good, while not using any heuristic.

Revised: Whether the aspect of bidirectional or the symbolic search contributes the most to this performance, we haven’t checked, but we expect one would need both.

Original: Recall that planning pattern databases are serving as heuristics  and that they are based on a complete backward exploration in some state-space abstraction.
Revised: SymbBidir performs much better in Agricola than the other planners. Using a PDB heuristics seems to hurt symbolic search and fails in 5 to 9 instances where SymbBiDir succeeds. Some of these problems have been identiﬁed and overcome by using perimeter pattern databases (Felner and Ofek 2007).

Revised: About the bad performance in Spider and Snake there are reasons on why the BDD explode , which relate to the subtle ordering problem of dependent BDD variables in grids, This issue has been analyzed and proven to be crucial for representing the goal to ConnectFour as a BDD (Edelkamp and

Revised: Kissmann 2011), and might be detected fully automatically.

Revised: Recall, that planning pattern databases are serving as heuristics and are based on a complete backward exploration in some state-space abstraction.

Original: Often a larger number of (hopefully) diverse and complementary patterns are generated and the corresponding databases sought to be combined in an admissible manner to preserve being a lower bound.
Revised: Often a larger number of (hopefully) diverse and complementary patterns are generated and the corresponding databases sought to be combined in an admissible manner to preserve being a lower bound.

Original: Perimeter pattern databases turned out to be good candidates in this PDB zoo (Felner and Ofek 2007). While Scorpion is an explicit-state PDB planner (performing slightly worse to SymbBiDir in IPC 2018), it showed distinguished performance, and further results illustrate that it is performing much better across all IPC benchmarks, i.e., ones including the ones from previous competitions (Seipp and Helmert 2018; 2019). 
Revised: While Scorpion is partly a PDB planner performing slightly worse to SymbBiDir in IPC 2018, it showed distinguished performance and scored best in 5 out of 10 domains. Further investigations illustrate that it is performing much better across all IPC benchmarks, i.e., ones including the ones from previous competitions (Seipp and Helmert 2018; 2019). It also has to be added that part of the success of Scorpion is due to Cartesian abstractions combined with a counterexample abstraction reﬁnement (CEGAR) approach (Clarke et al. 2000).

Revised: As both heuristics are based on state space abstractions, one may view PDB and Cartesian abstraction heuristics as being related estimates of a similar type.

Original:  When normalizing the success alongside the domains (some have 150 instances), Scorpion compares well with the top-tier symbolic planning systems: 0.621 (Complementary 2), 0.601 (Scorpion), 0.594 (Planning PDBs), 0.576 (Complementary 1), 0.555 (SymBiDir).
Revised: While PDBs and Cartesian abstractions are an important part of Scorpion, much of its strength is in the sophisticated method to combine these abstraction heuristics. While the competing BDD based PDB planners mainly use 0/1 cost partitioning a more advanced concept saturated cost-partitioning. When normalizing the success alongside the domains (some have 150 instances), Scorpion compares well with the top-tier symbolic planning systems: 0.621 (Complementary 2), 0.601 (Scorpion), 0.594 (Planning PDBs), 0.576 (Complementary 1), 0.555 (SymBiDir).

Original: Using Delﬁ in this comparison hardly applies, as for this case the training set overlaps the test set.
Revised: Using Delﬁ in this comparison hardly applies, as for this case the training set overlaps the test set.

Revised: Again, all or most planners, even the ones that were not at the top, had some positive results (e.g. being among the planners that solve most instances in some individual domains).

Revised: Even planners at the bottom have some cases where they perform among the best (e.g. ma-plan in Snake).

Original: Portfolio Planning
Revised: Portfolio Planning

Original: Portfolios erected on existing plan technology, are a recurring pattern in many competitions, and range from restarting strategies, over learning classiﬁers, to scheduling time slices to existing planners.
Revised: Portfolios erected on existing plan technology, are a recurring pattern in many competitions, and range from restarting strategies, over learning classiﬁers, to scheduling time slices to existing planners.

Original: Once having ﬁxed the metric and submission instructions, the IPC 2018 organizers felt that they had to follow them.
Revised: Once having ﬁxed the metric and submission instructions, the IPC 2018 organizers felt that they had to follow them.

Original: If a planner wins by the metric they decided on before, the competition and the organizers didn’t crown it the winner, the teams would rightfully complain.
Revised: If a planner wins by the metric they decided on before, the competition and the organizers didn’t crown it the winner, the teams would rightfully complain.

Original: The coverage metric of problems  solved seems not to be the issue, so one thing the organizers could have done, would have been to change the rules for submissions.
Revised: The coverage metric of problems being solved seemed not to be the issue, so one thing the organizers could have done, would have been to change the rules for submissions.

Original: They explained that they had an internal discussion about portfolios early on in the course of running the competition, but decided that the line between a planner with multiple components and a portfolio was too blurry to accurately deﬁne.
Revised: They explained that they had an internal discussion about portfolios early on in the course of running the competition, but decided that the line between a planner with multiple components and a portfolio was too blurry to accurately deﬁne.

Original: That is why they decided to not have a special rule against portfolios.
Revised: That is why they decided to not have a special rule against portfolios.

Original: As one reason given, the LAMA system (Richter and Westphal 2010), one previous IPC winner, might have to be considered a portfolio, because it runs different planners one after the other.
Revised: As one reason given, the LAMA system (Richter and Westphal 2010), one previous IPC winner, might have to be considered a portfolio, because it runs different planners one after the other.

Original: In terms of the organizers of the competition is difﬁcult to set rules that identify portfolios to give them a special treatment, because either they’d be too restrictive and most planners of the competition would be considered portfolios, or they would be too ambiguous, generating complains about what planners are considered to be portfolios. 
Revised: In terms of the organizers of the competition is difﬁcult to set rules that identify portfolios to give them a special treatment, because either they’d be too restrictive and most planners of the competition would be considered portfolios, or they would be too ambiguous, generating complains about what planners are considered to be portfolios. This said, there are certainly many interesting aspects to be learned from portfolios on a per-domain or even per-instance base.

Revised: Proper portfolio designs with a close-to-optimal choice of planners as in Delﬁ is a research area on its own.

Original: One way to limit the impact of portfolios in the competition is what Mauro and others have suggested for the IPC 2 , where planners are evaluated based on how well they do on individual instances/domains, rather than on getting a good average score.
Revised: One way to limit the impact of portfolios in the competition is what Mauro and others have suggested for the Sparkle planning competition 2 , where planners are evaluated based on how well they do on individual instances/domains, rather than on getting a good average score.

Original: However, this suggestion comes with some issues as well.
Revised: However, this suggestion comes with some issues as well.

Original: In particular, the score of a planner completely depends on which other planners are submitted to the competition.
Revised: In particular, the score of a planner completely depends on which other planners are submitted to the competition.

Original: Henceforth, if someone submits a version of your planner that works only slightly better, he could get 0 points.
Revised: Henceforth, if someone submits a version of your planner that works only slightly better, he could get 0 points.

Original: One has to wait for the results to see how the approach materializes.
Revised: One has to wait for the results to see how the approach materializes.

Original: Unfortunately, the organizers of this competition are only running the agile track, without insisting on cost-optimal plans.
Revised: Unfortunately, the organizers of this competition are only running the agile track, without insisting on cost-optimal plans.

Original: In complexity terms, optimization is known to be of crucial importance.
Revised: In complexity terms, however, optimality is known to be of crucial importance.

Original: For example, solving many planning benchmarks (such as Blocksworld, Logistics, Sliding Tile Puzzle) are polynomial, if —as the case in satisﬁcing planning— only plan existence is considered, while  optimization is hard (Helmert 2008).
Revised: For example, ﬁnding any plan for many planning benchmarks (such as Blocksworld, Logistics, SlidingTile Puzzle) is polynomial. In this case of satisﬁcing planning where only plan existence is requested problem tend to be tractable, while the corresponding optimization are often provably hard (Slaney and Thi´ebaux 2001; Parberry 2015; Helmert 2008).

Original: The emerging set of portfolio  may be seen as a side effect of the requirement of releasing source code, as it becomes easier to bundle the planners into one code base, but access to the source code is not a strict necessity for these type of planners.
Revised: The emerging set of portfolio and the difﬁculty of excluding them may be seen as a side effect of the requirement of releasing source code for the planners, as it becomes easier to bundle the planners into one code base. Of course public access to the source code is not a strict necessity for these type of planners.

Original: For some people, the core issue and concern of portfolio planning is that other researchers use their code, and not so much that the participating planner is a portfolio.
Revised: For some planning researchers, the core issue and concern of portfolio planning is that other researchers use their code, and not so much that the participating planner is a portfolio.

Original: So the organizers thought about having had a rule against using code from another research groups.
Revised: So the organizers thought about having had a rule against using code from another research groups.

Original: That would have excluded most planners based on Fast Downward planner framework, though, and since they were the majority of the submissions, this would not have been a good idea as well.
Revised: That would have excluded most planners based on Fast Downward planner framework, though, and since they were the majority of the submissions, this would not have been a good idea as well.

Original: The solution Fahiem Bacchus suggested (in personal conversation with the author) based on his own experience with portfolios in the SAT competitions, is having stated a license that prohibits the use of the code in other tools, would be an option, but then the authors of the planners would have to do so before the competition.
Revised: The solution Fahiem Bacchus suggested (in personal conversation with the authors) based on his own experience with portfolios in the SAT competitions, is having stated a license that prohibits the use of the code in other tools, would be an option, but then the authors of the planners would have to do so before the competition.

Original: In case of planners based on Fast
Revised: In case of planners based on Fast

Original: Downward, this, however, is also problematic because such a clause would be hardly compatible with the license of the framework.
Revised: Downward, this, however, is also problematic because such a clause would be hardly compatible with the license of the framework.

Original: Whileintuitively rather obvious, it is far from simple to distinguish portfolio from non-portfolio planner in a formal deﬁnition.
Revised: Whileintuitively rather obvious, it is far from simple to distinguish portfolio from non-portfolio planner in a formal deﬁnition.

Original: One may try to start with the following criteria.
Revised: One may try to start with the following criteria.

Original: A planner portfolio selects, invokes, and possibly terminates different existing planners, based on a trained or hard-coded decision procedure.
Revised: A planner portfolio selects, invokes, and possibly terminates different existing planners, based on a trained or hard-coded decision procedure.

Original: These deﬁnition may not be a perfect discriminator, as one might be able to transform a portfolio into a non-portfolio without changing the performance by much: just moving the decision procedure further down the line.
Revised: This deﬁnition may not be a perfect discriminator, as one might be able to transform a portfolio into a non-portfolio without changing the performance by much: just moving the decision procedure further down the line.

Original: It does also not cover a planner that uses the maximum of ten heuristics in an A* search.
Revised: It does also not cover a planner that uses the maximum of ten heuristics in an A* search.

Original: Some people would like to treat this as a portfolio, because there is no contribution except for the selection  of the ten heuristics.
Revised: Some people would like to treat this as a portfolio, because there is no contribution except for the selection procedure of the ten heuristics.

Original: At the end, the question remains on when a planner is a novel contribution.
Revised: At the end, the question remains on when a planner is a novel contribution.

Original: Another suggested deﬁnition for identifying portfolio planning is the following.A
Revised: Anothersuggested deﬁnition for identifying portfolio planning is the following.

Original: non- portfolio planner is a single core planning tech nology which invokes 
Revised: A non-portfolio planner is a single core planning technology, which invokes a plan search in one state space.

Original: a plan search in one state space. But what is with traversing state-space abstractions, which are needed to compute heuristic estimates?
Revised: But what is with traversing state-space abstractions, which are needed to compute heuristic estimates?

Original: Clearly,  deﬁning portfolios turns out to be intrinsically difﬁcult.
Revised: Clearly, as highlighted by the IPC organizers, deﬁning portfolios turns out to be intrinsically difﬁcult.

Original: There are planners that are clearly portfolios, there are planners that are clearly not, and there is a larger gray area in between.
Revised: There are planners that are clearly portfolios, there are planners that are clearly not, and there is a larger gray area in between.

Original: According to a deﬁnition, FF (Hoffmann and Nebel 2001) should not be judged as a portfolio planner.
Revised: According to a deﬁnition, FF (Hoffmann and Nebel 2001) should not be judged as a portfolio planner.

Original: It searches one state space with one heuristic.
Revised: It searches one state space with one heuristic.

Original: But FF switches from enforced hill-climbing to best-ﬁrst search, based on some progress measure.
Revised: But FF switches from enforced hill-climbing to best-ﬁrst search, based on some progress measure.

Original: This alone should not classify it as a portfolio approach.
Revised: This alone should not classify it as a portfolio approach.

Original: LAMA runs a greedy search based on h FF and a landmark heuristic (three techniques developed by different authors) and then several weighted A* searches (a different planner and an algorithm also developed by other authors).
Revised: LAMA runs a greedy search based on h FF and a landmark heuristic (three techniques developed by different authors) and then several weighted A* searches (a different planner and an algorithm also developed by other authors).

Original: LAMA may or may not be seen as a portfolio.
Revised: LAMA may or may not be seen as a portfolio.

Original: If it runs three independent searches in parallel, than this may be interpreted as a portfolio, but the interconnection of the search is more subtle.
Revised: If it runs three independent searches in parallel, then this may be interpreted as a portfolio technology, but the interconnection of the search is more subtle.

Original: LAMA had additional algorithmic contributions on how to move back and forth the states in the different priority queues.
Revised: LAMA had additional algorithmic contributions on how to move back and forth the states in the different priority queues.

Original: If LAMA continues searching the same[Table] search space, this is a sign of a non-portfolio.
Revised: If LAMA continues searching the same search space, this is a sign of a non-portfolio.

Original: At least it does not start different existing planner.
Revised: At least it does not start different existing planner.

Original: It is, however, abundantly clear that Delﬁ is a portfolio planner (not even the authors questions that).
Revised: It is, however, abundantly clear that Delﬁ is a portfolio planner (not even the authors questions that).

Original: It even prints its task-dependent calls to the planner binaries.
Revised: It even logs its task-dependent calls to the planner binaries.

Original: Delﬁ actually uses 2 executables, SymBA* and 16 parameters of selecting planners in the Fast Downward framework.
Revised: Delﬁ actually uses 2 executables, SymBA*, and 16 parameters of selecting planners in the Fast Downward framework.

Original: It has a decision procedure trained on a set of manual selected planning tasks.
Revised: It has a decision procedure trained on a set of manual selected planning tasks.

Original: Note that in this setting, we do not count the learning as running but  programming time.
Revised: Note that in this setting, we do not count the learning as running, but as programming time.

Original: The belated aspect of other people that contribute new planning ideas to the IPC is that  more than 99.9% of the actual running time  is exclusively spend on existing technology.
Revised: The performance overhead of portfolio designs can be small. The often criticized effect is that frequently more than 99.9% of the actual running time of a portfolio planner is exclusively spend on existing technology.

Revised: This is a probably unwanted aspect, which can makes other competitors that contribute non-portfolio planners wondering and reluctant to tune their planners for efﬁciency.

Revised: Of course, the size of a contribution must not necessarily be taken in direct correspondence to the proﬁled time that was spend in running the code added.

Original: By public access to the planners at IPC 2018, one can look at the source code of the contributed planner to validate, on whether or not a planner is in fact a portfolio.
Revised: By public access to the planners at IPC 2018, one can look at the source code of the contributed planner to validate, on whether or not a planner is in fact a portfolio.

Original: Fast Downward’s code base has grown too big, there are pros and cons to that.
Revised: Fast Downward’s code base has grown too big, there are pros and cons to that.

Original: On the pro side the planner suite is good for benchmarking.
Revised: On the pro side the planner suite is good for benchmarking.

Original: For the symbolic search engines in
Revised: For the symbolic search engines in

Original: IPC 2018, it was better to use it to combine explicit with symbolic search than sticking to an independent technology in Gamer.
Revised: IPC 2018, it was better to use it to combine explicit with symbolic search than sticking to an independent technology in Gamer.

Original: In fact, there are myriads of parameters that make
Revised: In fact, there are myriads of parameters that make

Original: Fast Downward behave totally different.
Revised: Fast Downward behave totally different.

Original: Fast Downward is no longer one planner, it is a framework.
Revised: Fast Downward is no longer one planner, it is a framework.

Original: On the cons side, results on mixing different calls it may blur the messages you to take home.
Revised: On the cons side, results on mixing different calls it may blur the messages you to take home.

Original: There were at least two different portfolio planners in IPC 2018:
Revised: There were at least two different portfolio planners in IPC 2018:

Original: Delﬁ1 and Delﬁ2, where Delﬁ1 was so much better, so that in the following we concentrate on this one, and used
Revised: Delﬁ1 and Delﬁ2, where Delﬁ1 was so much better, so that in the following we concentrate on this one, and used

Original: Delﬁ for its shortcut notation.
Revised: Delﬁ for its shortcut notation.

Original: There is published work of the planner authors in the IPC booklet that explain the architecture and the machine learning approach of using deep neural nets in more detail, so that we concentrate on the main aspects.
Revised: There is published work of the planner authors in the IPC booklet that explain the architecture and the machine learning approach of using deep neural nets in more detail, so that we concentrate on the main aspects.

Original: The main idea is to train a classiﬁer on the performance curves of known planning benchmark problems, provided as input images.
Revised: The main idea is to train a classiﬁer on the performance curves of known planning benchmark problems, provided as input images.

Original: We had some problems to reproduce the results on our machine, but could look at the competition results.
Revised: We had some problems to reproduce the results on our machine, but could look at the competition results.

Original: The planners being invoked by Delﬁ in the IPCare shown in Table 2.
Revised: The planners being invoked by Delﬁ in the IPCare shown in Table 2.

Revised: Note that Delﬁ combines the heuristics listed with symmetry and partial-order reduction.

Original: The story on portfolios will go on.
Revised: The story on portfolios will go on.

Revised: Portfolios have already started integrating the systems from 2018.

Original: Essentially, even when pushing the ﬁeld with new and brilliant ideas, one hardly can win the race against a portfolio, if there is no ban.
Revised: Essentially, even when pushing the ﬁeld with new and brilliant ideas, one hardly can win the race against a portfolio, at least in general terms.

Original: Portfolios have already started integrating the systems from 2018.
Revised: While portfolios have dominated some tracks in the IPC (the satisﬁcing track, for example) in the optimal track, the winner of IPC 2014 was not a portfolio. As seen in IPC 2018, there were several planners that got a very close performance to Delﬁ. Also, some other portfolios participated. Delﬁ2 and other portfolios (MSP and DecStar could be considered port-

Original:  What is worse, with winning of the IPC in the pocket, portfolio planners help to acquire project money and to publish on high-ranked journals conferences, where nonportfolio planners  have a hard time arguing that they are carrying the actual contribution inplanning technology, as  a second place is  not considered state-of-the-art.
Revised: [Table] folios as well) were behind many non-portfolio planners. Overall, the results do not show dominance of portfolio planners in general. What is worse, with winning of the IPC in the pocket, portfolio planners help to acquire project money and to publish in high-ranked journals and conference proceedings, where non-portfolio planners often have a harder time arguing that they are carrying the actual contribution in technology, as especially in research, a second place is often not considered state-of-the-art.

Revised: Instead of arguing, whether portfolios should participate or not, we should discuss about how people interpret the results of the IPC and how an analysis that goes beyond Planner X is the winner is absolutely necessary.

Revised: Even planners that are not at the top show that there are domains where they can be really useful.

Revised: We view this position paper as one step towards this end.

Original: About credits.
Revised: About credits.

Original: For a scientiﬁc paper it is rather clear that one has to become a co-author, if one contributes substantially to the outcome.
Revised: For a scientiﬁc paper it is rather clear that one has to become a co-author, if one contributes substantially to the outcome.

Original: With portfolios this is slightly different.
Revised: With portfolios this is slightly different.

Original: In the extreme case,  the one  contributes and the other ones take the credits for the  work.
Revised: In the extreme case, it may happen tbat the one coder contributes and the other one coder using portfolio technology take the credits for the efﬁciencies of the work.

Original: Of course we wouldn’t ask Hart, Nilsson and Raphael to be co-authors of every forward-search planning paper using
Revised: Of course we wouldn’t ask Hart, Nilsson and Raphael to be co-authors of every forward-search planning paper using

Original: A* but we would still cite their work (Hart, Nilsson, and
Revised: A* but we would still cite their work (Hart, Nilsson, and

Original: Raphael 1972).
Revised: Raphael 1972).

Original: Probably the same is true for portfolio planners: they should give credit, where it is due (and the planner abstracts do this).
Revised: Probably the same is true for portfolio planners: they should give credit, where it is due (and the planner abstracts do this).

Original: They also have their own contribution.
Revised: No question, portfolios also have their own contribution.

Original: The contribution in a portfolio planner is the combination of techniques, e.g., how much better is this combination than just running all n components for 1 n -th of the time.
Revised: The contribution in a portfolio planner is the combination of techniques, e.g., how much better is this combination than just running all n components for 1 n -th of the time.

Original: In this respect the competition booklet helps a lot as it links the IPC planners to the  people, but it is rarely counted  a publication.
Revised: In this respect the competition booklet helps a lot as it links the IPC planners to the outside people, but one the other hand, in the scientiﬁc race a booklet is no archival publication is rarely counted as a success.

Original: This is what one may ask for a portfolio,  be explicit on which planner call achieves which individual performance, and not to bury this information in a lot of other stuff, e.g., on how advanced the  neural network training is.
Revised: This is what one may ask for a portfolio, to be explicit on which planner call achieves which individual performance, and not to bury this information in a lot of other stuff, e.g., on how advanced the machine learning (e.g., deep neural network training) is.

Original: Often we insist on a proper publication before the release of the code, but this  does not work for competitions like IPC 2018.
Revised: We often insist on a proper publication before the release of the code, but this also does not work for competitions like IPC 2018.

Original: The problem is essential, as with the competition the coders provide all source to the public, so we should take more care on who contributes what.
Revised: The problem is essential, as with the competition the coders provide all source to the public, so we should take more care on who contributes what.

Original: If an outside contribution is dominating the own one consider asking the authors.
Revised: If an outside contribution is dominating the own one consider asking the authors.

Original: Sometimes you cite and acknowledge, sometimes you feel this is authorship.
Revised: Sometimes you cite and acknowledge, sometimes you feel this is authorship.

Original: For the IPC, we see people taking code, shake it a bit to improve the results slightly, and go on publishing.
Revised: For the IPC, we see people taking code, shake it a bit to improve the results slightly, and go on publishing.

Original: Whether or not portfolio planners being trained on sample plans are domain-independent, is also a controversy, especially given that training plan samples selected by hand.
Revised: Whether or not portfolio planners being trained on sample plans are domain-independent, is also a controversy, especially given that training plan samples selected by hand.

Original: Extremist think they are not, but other people may think differently.
Revised: Extremist think they are not, but other people may think differently.

Original: Surely portfolio planners belong to the learning track.
Revised: Surely portfolio planners belong to the learning track.

Original: Of course, organizers were quite happy to have that many competitors, and for us it was a tremendous success to see how good our planners performs, even when facing portfolios.
Revised: Of course, organizers were quite happy to have that many competitors, and for us it was a tremendous success to see how good our planners performs, even when facing portfolios.

Original: We enjoyed to see how hard it is to get some good result in cost-optimal by machine learning.
Revised: We enjoyed to see how hard it is to get some good result in cost-optimal by machine learning.

Original: Conclusion
Revised: Conclusion

Revised: This position paper aims at arguing on what is the currently leading technology and discusses whether or not portfolios help to push or blur the outcome of a competition.

Revised: There is indication but not one deﬁnite conclusion alias strong proposal along this paper on how to deal with the given observations, the main purpose of the paper is look behind the scenes and to spawn a discussion.

Revised: Such discussions have tradition in the IPC.

Revised: We had a discussion on domain-independence at the emerge of control rules in TL- and TAL-Plan (Bacchus and Kabanza 2000;

Revised: Kvarnstr¨om and Magnusson 2003), we also had a discussion on the effect of hand-coded selection in planners like SGPlan (Wah and Chen 2004), with complaints on hand-written domain-dependent branching inside the planners’ code.

Original:  The international planning competition 2018 pushed the ﬁeld in action planning, set up and executed a well-designed externally controlled experiment, aimed at insights about the true performance of planners, falsiﬁed and strengthened hypotheses on essential components, compared different technologies on a common rule set, same architecture, and an agreed input formalism.
Revised: Interestingly, the one who complains most about the current IPC organizes the next IPC. Now in 2018 the topic is portfolio planning and the problem of identifying and securing individual planner contributions. The international planning competition 2018 pushed the ﬁeld in action planning, set up and executed a well-designed externally controlled experiment, aimed at insights about the true performance of planners, falsiﬁed and strengthened hypotheses on essential components, compared different technologies on a common rule set, same architecture, and an agreed input formalism.

Original: It awarded scientiﬁc prizes and provided opportunities for upcoming publications.
Revised: It awarded scientiﬁc prizes and provided opportunities for upcoming publications.

Original: The evaluation is much better than what one experienced in conference and journal papers.
Revised: The evaluation is much better than what one experiences in conference and journal papers.

Original: The results are often surprising, when compared to  wisdomonly taken fromlooking at existing publications.
Revised: The results are often surprising, when compared to the wisdom taken from existing publications.

Original: Of course, every competition is limited in what it can prove, but its scientiﬁc impact should not to be underestimated.
Revised: Of course, every competition is limited in what it can prove, but its scientiﬁc impact is not to be underestimated.

Revised: This paper discusses two separate claims: 1) An analysis of the result of the optimal track, claiming that symbolic search and PDBs are leading methods for cost-optimal planning. 2)

Revised: The advantages and disadvantages of considering portfolios as part of the IPC.

Original: Inacompetition is always good to refer to a wider set of planners, but fundamental differences should be highlighted and could have been put into the awarding considerations.
Revised: The IPC has always been a competition where the best mix of scientiﬁc and engineering skill wins. It has never been just one good idea that won the competition, but also the skill to implement it efﬁciently, and even before portfolios there was a chance that the better software developer outperforms the better researcher, possibly even with something that would never be published. In this case most arguments are made only about IPC(Delﬁ vs Complementary/PlanningPDBs). The discussion of what is the place of portfolios in the IPC and other competitions tracks and events is of course a a general one.

Original:  It is ﬁne to have portfolios inside the competition, but they should at least be tagged as such, given a portfolio is a different type of planner and, otherwise, wrong conclusions are drawn from the event.
Revised: What to do? In a competition it is always good to refer to a wider set of planners, but fundamental differences should be highlighted and could have been put into the awarding considerations. It is ﬁne to have portfolios inside the competition, but they should at least be tagged as such, given a portfolio is a different type of planner, and, otherwise, wrong conclusions might be drawn from the event.

Original: Otherwise, the competition is doomed to swallow its own core contributors.
Revised: Otherwise, the competition is doomed to swallow its own core contributors.

Original: The risk is that these efforts will die out.
Revised: The risk is that these efforts will die out.

Original: Considering on how many different planning heuristics have been suggested in the past, given that places 2 toare symbolic search and planning pattern database planners only, and that the winner of IPC 2018 called a symbolic search planner half of its time is a striking fact.
Revised: Considering on how many different planning heuristics have been suggested in the past, given that places two to six are symbolic search and planning pattern database planners only, and that the winner of IPC 2018 called a symbolic search planner half of its time is a striking fact.

Original: We are not aware of any technology that performs better especially on the 2018 IPC benchmark set.
Revised: We are not aware of any technology that performs better especially on the 2018 IPC benchmark set.

Original: Given ﬂuctuations in the results many planners are playing in the same ballpark.
Revised: Given ﬂuctuations in the results many planners are playing in the same ballpark.

Original: Recent improvements and simpliﬁcations indicate that one can lift the results of symbolic pattern database planning towards winning the IPC 2018 competition post mortem. 
Revised: Recent improvements and simpliﬁcations indicate that one can lift the results of symbolic pattern database planning towards winning the IPC 2018 competition post mortem. While portfolios usually dominate non-optimal IPC tracks, it is indeed a tighter race in the cost-optimal track.

Original: Acknowledgement Programming is a serious job, but comes with  a lot of fun.
Revised: Optimal planning seems to be a tough nut to crack for portfolios given the limits in time and space, even when having a bigger toolbox. Portfolios can take on novel contribution for free and quite quickly. The advance one to put on top from event to event is counterbalanced with using many planners at once. Whether such an advance is always possible, is a subject projection.

Revised: But at least it may be argued that it appears that it might be exhausting for the competitors with non-portfolio planners to come up with novel, original and breakthrough technology at every new IPC and compete with the portfolios of the last one.

Original:  The IPC 2018 is a programming contest that allowed all competitors to impress with stunning and outstanding performance results on yet unseen complex, and diverse problem domains.
Revised: Acknowledgement Programming is a serious art and comes with a lot of fun. The IPC 2018 is a programming contest that allowed all competitors to impress with stunning and outstanding performance results on yet unseen complex, and diverse problem domains.

Original: We thank the competitors for the variety of new planning approaches given, advancing the state-of-the-art in many respects.
Revised: We thank all competitors for the variety of new planning approaches, advancing the stateof-the-art in many respects.

Original: The organizers of the IPC 2018 did a great job, both with the choice and design of the benchmark domains and for running the competition with that much care.
Revised: The organizers of the IPCdid a great job, both with the choice/design of the benchmark domains and for running the competition.

Original: Eventually, they had to decide on the winner according to the rules set.
Revised: Eventually, they had to decide on the winner according to the rules set