Original: Online Saturated Cost Partitioning 
Revised: Online Saturated Cost Partitioning for Classical Planning

Original: Abstract
Revised: Abstract

Original: Saturated cost partitioning is a general method for admissibly adding heuristic estimates for optimal state-space search.
Revised: Saturated cost partitioning is a general method for admissibly adding heuristic estimates for optimal state-space search.

Original: The algorithm strongly depends on the order in which it consid- ers the heuristics.
Revised: The algorithm strongly depends on the order in which it consid- ers the heuristics.

Original: The strongest previous approach precomputes a set of diverse orders and the corresponding saturated cost partitionings before the search.
Revised: The strongest previous approach precomputes a set of diverse orders and the corresponding saturated cost partitionings before the search.

Original: This makes evaluating the overall heuristic very fast, but requires a long precompu- tation phase.
Revised: This makes evaluating the overall heuristic very fast, but requires a long precompu- tation phase.

Original: By diversifying the set of orders online during the search we drastically speed up the planning process and even solve slightly more tasks.
Revised: By diversifying the set of orders online during the search we drastically speed up the planning process and even solve slightly more tasks.

Original: Saturated Cost Partitioning
Revised: Saturated Cost Partitioning

Original: One of the main approaches for solving classical planning tasks optimally is using the A ∗ algorithm (Hart, Nilsson, and
Revised: One of the main approaches for solving classical planning tasks optimally is using the A ∗ algorithm (Hart, Nilsson, and

Original: Raphael 1968) with an admissible heuristic (Pearl 1984).
Revised: Raphael 1968) with an admissible heuristic (Pearl 1984).

Original: Since a single heuristic usually fails to capture enough details of the planning task, it is often beneﬁcial to compute multiple heuristics and to combine their estimates (Holte et al. 2006).
Revised: Since a single heuristic usually fails to capture enough details of the planning task, it is often beneﬁcial to compute multiple heuristics and to combine their estimates (Holte et al. 2006).

Original: The preferable method for admissibly combining heuristic estimates is cost partitioning (Haslum, Bonet, and Geffner 2005; Haslum et al. ; Katz and Domshlak 2008; 2010; Pommerening, R¨oger, and Helmert 2013).
Revised: The preferable method for admissibly combining heuristic estimates is cost partitioning (Haslum, Bonet, and Geffner 2005; Haslum et al. 2007; Katz and Domshlak 2008, 2010; Pommerening, R¨oger, and Helmert 2013).

Original: By distributing the original costs among the heuristics, cost partitioning makes the sum of heuristic estimates (under the reduced cost functions) admissible.
Revised: By distributing the original costs among the heuristics, cost partitioning makes the sum of heuristic estimates (under the reduced cost functions) admissible.

Original: Saturated cost partitioning (SCP) is one of the strongest methods for ﬁnding cost partitionings (Seipp and Helmert 2014; Seipp, Keller, and Helmert 2020).
Revised: Saturated cost partitioning (SCP) is one of the strongest methods for ﬁnding cost partitionings (Seipp and Helmert 2014; Seipp, Keller, and Helmert 2020).

Original: At the core of the
Revised: At the core of the

Original: SCP algorithm lies the insight that we can often reduce the (action) cost function of a planning task and still obtain the same heuristic estimates.
Revised: SCP algorithm lies the insight that we can often reduce the (action) cost function of a planning task and still obtain the same heuristic estimates.

Original: This notion is captured by so-called saturated cost functions.
Revised: This notion is captured by so-called saturated cost functions.

Original: An (action) cost function scf is saturated for a heuristic h , an original cost function cost and a subset S (cid:48) of states in the planning task, if scf ( a ) ≤ cost ( a ) for each action a and for all states s ∈ S (cid:48)
Revised: An (action) cost function scf is saturated for a heuristic h , an original cost function cost and a subset S (cid:48) of states in the planning task, if scf ( a ) ≤ cost ( a ) for each action a and for all states s ∈ S (cid:48)

Original: the heuristic estimate by h for s is the same regardless of whether we evaluate h under cost or scf .
Revised: the heuristic estimate by h for s is the same regardless of whether we evaluate h under cost or scf .

Original: We call a function that computes a saturated cost function for a given heuristic and cost function a saturator .
Revised: We call a function that computes a saturated cost function for a given heuristic and cost function a saturator .

Original: The original formulation assumed S (cid:48) to always be the set of all states.
Revised: Algorithm 1 shows how the SCP procedure computes saturated cost functions that form a cost partitioning of a given cost function cost over an ordered sequence of heuristics ω .

Original: This deﬁnition
Revised: The algorithm starts by computing a saturated cost function

Original: 1: function S ATURATED C OST P ARTITIONING ( ω , cost ) 2:
Revised: 1: function S ATURATED C OST P ARTITIONING ( ω , cost ) 2:

Original: C ← (cid:104)(cid:105) 3: for all h ∈ ω do 4: scfh ← saturate h ( cost ) 5: C ← C ⊕ scf h 6: cost  ← cost  − scf h 7: return C has been generalized recently to allow preserving the estimates for a subset of states and we use the perim (cid:63) saturator for choosing the subsets here, because it yields the strongest heuristics (Seipp and Helmert 2019). Algorithm 1 shows how the SCP procedure computes saturated cost functions that form a cost partitioning of a given cost function cost over an ordered sequence of heuristics ω . The algorithm starts by computing a saturated cost function for the ﬁrst heuristic h in ω , i.e., it lets a saturator saturate h(which is ﬁxed to be perim (cid:63) here) compute the fraction of the action costs that are needed to preserve the estimates by h for a subset of states under the original cost function (line 4).
Revised: C ← (cid:104)(cid:105) 3: for all h ∈ ω do 4: scf ← saturate h ( cost ) 5: append scf to C 6: cost ( a ) ← cost ( a ) − scf ( a ) for all actions a 7: return C for the ﬁrst heuristic h in ω , i.e., it lets a saturator saturate h compute the fraction of the action costs that are needed to preserve the estimates by h for a subset of states under the original cost function (line 4).

Original: Afterwards, it iteratively subtracts the costs given to h from the original costs (line 6) and considers the next heuristic until all heuristics have been treated this way.
Revised: Afterwards, it iteratively subtracts the costs given to h from the original costs (line 6) and considers the next heuristic until all heuristics have been treated this way.

Original: The sequence of computed saturated cost functions forms the resulting cost partitioning C .
Revised: The sequence of computed saturated cost functions forms the resulting cost partitioning C .

Original: We write h SCP ω for the cost partitioning heuristic that results from applying the SCP algorithm to the heuristic order ω .
Revised: We write h SCP ω for the cost partitioning heuristic that results from applying the SCP algorithm to the heuristic order ω .

Original: The quality of a SCP heuristic greatly depends on the order in which the heuristics are considered.
Revised: The original SCP formulation assumed S (cid:48) to always be the set of all states.

Original: In this work, we  use  the greedy ordering method withthe h stolen scoring function, the best ordering in previous work on saturated cost partitioning (Seipp, Keller, and Helmert 2020).
Revised: This deﬁnition has been generalized recently to allow preserving the estimates for a subset of states, giving rise to new saturator types (Seipp and Helmert 2019). One of the new saturators is perim , which preserves the estimates of all states within a given perimeter of the goal. For example, for a given heuristic h and a state s we can use perim to preserve the heuristic estimates of all states s (cid:48) with h ( s (cid:48) ) ≤ h ( s ) (and reduce all higher estimates to h ( s ) ).

Original: As in previous work on SCP, we focus on abstraction heuristics (Helmert, Haslum, and Hoffmann 2007). Ofﬂine Diversiﬁcation of SCP Heuristics Most of the previous work on the topic precomputes SCPs ofﬂine , i.e., before the search and then computes  the maximum over the 
Revised: The perim saturator often yields higher estimates for a given state than the all saturator, which preserves all estimates. However, perim also often ignores costs that could be used to improve the heuristic estimates of other states. Therefore, the strongest method by Seipp and Helmert (2019), perim (cid:63) , ﬁrst computes a saturated cost partitioning using perim and then uses the remaining costs to compute a saturated cost partitioning that preserves all estimates under the remaining cost function.

Original:  SCP heuristic estimates for a given state during the search. Algorithm 2 shows the strongest ofﬂine SCP algorithm from
Revised: We use perim (cid:63) in all experiments below. The quality of an SCP heuristic greatly depends on the order in which the heuristics are considered. In this work, we use the greedy ordering method with the h stolen scoring function, the best ordering in previous work on saturated

Original: Algorithm 2 Ofﬂine diversiﬁcation.
Revised: Algorithm 2 Ofﬂine diversiﬁcation.

Original: Find a diverse set of heuristic orders Ω for SCP before the search.
Revised: Find a diverse set of heuristic orders Ω for SCP before the search.

Original: 1: function O FFLINE D IVERSIFICATION 2: Ω ← ∅ 3: ˆ S ← sample 1000 states 4: repeat 5: s ← sample state 6: ω ← greedy order for s 7: if ∃ s (cid:48)
Revised: 1: function O FFLINE D IVERSIFICATION 2: Ω ← ∅ 3: ˆ S ← sample 1000 states 4: repeat 5: s ← sample state 6: ω ← greedy order for s 7: if ∃ s (cid:48)

Original: ∈
Revised: ∈

Original: ˆ S : h SCP ω ( s (cid:48) ) > sup ω (cid:48) ∈
Revised: ˆ S : h SCP ω ( s (cid:48) ) > sup ω (cid:48) ∈

Original: Ω h SCP ω (cid:48)
Revised: Ω h SCP ω (cid:48)

Original: ( s (cid:48) )
Revised: ( s (cid:48) )

Original: then 8: Ω ← Ω ∪ { ω } 9: until time spent in function ≥ T 10: return Ω the literature.
Revised: then 8: Ω ← Ω ∪ { ω } 9: until time spent in function ≥ T 10: return Ω cost partitioning (Seipp, Keller, and Helmert 2020).

Revised: For each heuristic h and a given state s , it computes the fraction of h ( s ) over the costs “stolen” by h , i.e., the amount of costs that h wants to steal from other heuristics for preserving its estimates.

Revised: Then the greedy method orders heuristics by their h stolen fractions in decreasing order.

Revised: As in previous work on

Revised: SCP, we focus on abstraction heuristics (Helmert, Haslum, and Hoffmann 2007).

Revised: Ofﬂine Diversiﬁcation of SCP Heuristics Most of the previous work on the topic precomputes SCPs ofﬂine , i.e., before the search and then computes the maximum over the

Original:  It samples 1000 states ˆ S with random walks (line 3) and then iteratively samples a new state s (line 5), computes a greedy order ω for s (line 6) and keeps ω if  h SCP ω yields a higher heuristic estimate for any of the samples in ˆ
Revised: SCP heuristic estimates for a given state during the search. Algorithm 2 shows the strongest ofﬂine SCP algorithm from the literature (Seipp, Keller, and Helmert 2020). It samples 1000 states ˆ S with random walks (line 3) and then iteratively samples a new state s (line 5), computes a greedy order ω for s (line 6) and keeps ω if it is diverse , that is, h SCP ω yields a higher heuristic estimate for any of the samples in ˆ

Original: S than all previously stored orders (lines 7–8).
Revised: S than all previously stored orders (lines 7–8).

Original: (The supremum of the empty set is −∞ .)
Revised: (The supremum of the empty set is −∞ .)

Original: The ofﬂine diversiﬁcation procedure stops and returns the found set of orders Ω after reaching a given time limit.
Revised: The ofﬂine diversiﬁcation procedure stops and returns the found set of orders Ω after reaching a given time limit.

Original: This last characteristic is the main drawback of the algorithm: the A ∗ search can only start after the ofﬂine diversiﬁcation ﬁnishes and so far there is no good stopping criterion except for a ﬁxed time limit.
Revised: This last characteristic is the main drawback of the algorithm: the A ∗ search can only start after the ofﬂine diversiﬁcation ﬁnishes and so far there is no good stopping criterion except for a ﬁxed time limit.

Original: Seipp,
Revised: Seipp,

Original: Keller, and Helmert (2020) showed that a limit of 1000 seconds leads to solving the highest number of IPC benchmarks in 30 minutes, but such a high time limit obviously bloats the solving time for many tasks, especially for those that blind search would solve instantly.
Revised: Keller, and Helmert (2020) showed that a limit of 1000 seconds leads to solving the highest number of IPC benchmarks in 30 minutes, but such a high time limit obviously bloats the solving time for many tasks, especially for those that blind search would solve instantly.

Original: Online Computation of SCP Heuristics Instead of precomputing SCP heuristics before the search, we can also compute them online , i.e., during the search.
Revised: Online Computation of SCP Heuristics Instead of precomputing SCP heuristics before the search, we can also compute them online , i.e., during the search.

Original: This approach, which we call online-nodiv , computes a greedy order and the corresponding SCP heuristic for each state evaluated during the search.
Revised: This approach, which we call online-nodiv , computes a greedy order and the corresponding SCP heuristic for each state evaluated during the search.

Original: By design, online-nodiv can start the A ∗ search immediately and it has access to the states that are actually evaluated by A ∗ and not only to randomly sampled states like the ofﬂine diversiﬁcation procedure.
Revised: By design, online-nodiv can start the A ∗ search immediately and it has access to the states that are actually evaluated by A ∗ and not only to randomly sampled states like the ofﬂine diversiﬁcation procedure.

Original: As a result, the online-nodiv method has been shown to work well for landmark heuristics (Seipp, Keller, and Helmert2017).
Revised: As a result, the online-nodiv method has been shown to work well for landmark heuristics (Seipp, Keller, and Helmert

Revised: 1: function C OMPUTE H EURISTIC ( Ω , s ) 2: if S ELECT ( s ) and time spent in function < T then 3: ω ← greedy order for s 4: if h SCP ω ( s ) > sup ω

Revised: ∈ Ω h SCP ω ( s ) then 5: Ω ← Ω ∪ { ω } 6: return

Original:  However, computing an SCP over abstraction heuristics for each evaluated state slows down the heuristic evaluation so much that the online variant solves much fewer tasks than precomputed 
Revised: max ω ∈ Ω h SCP ω ( s ) 2017). However, computing an SCP over abstraction heuristics for each evaluated state slows down the heuristic evaluation so much that the online variant solves much fewer tasks than precomputed SCP heuristics (Seipp, Keller, and

Original: SCP heuristics (Seipp, Keller, and Helmert 2020). 
Revised: Helmert 2020). This kind of result is typical for optimal classical planning: more work per evaluated state often results in better estimates but does not outweigh the slower evaluation speed (e.g., Karpas, Katz, and Markovitch 2011; Seipp, Pommerening, and Helmert 2015).

Original: Online Diversiﬁcation of SCP Heuristics
Revised: Online Diversiﬁcation of SCP Heuristics

Original: In this work, we combine ingredients of the ofﬂine and online-nodiv variants to obtain the beneﬁts of both, i.e., fast solving times and high total coverage.
Revised: In this work, we combine ingredients of the ofﬂine and online-nodiv variants to obtain the beneﬁts of both, i.e., fast solving times and high total coverage.

Original: More precisely, we interleave heuristic diversiﬁcation and the A ∗ search: for a subset of the evaluated states, we compute a greedy order and store the corresponding SCP heuristic if it yields a more accurate estimate for the state at hand than all previously stored SCP heuristics.
Revised: More precisely, we interleave heuristic diversiﬁcation and the A ∗ search: for a subset of the evaluated states, we compute a greedy order and store the corresponding SCP heuristic if it yields a more accurate estimate for the state at hand than all previously stored SCP heuristics.

Original: Algorithm 3 shows pseudo-code for the approach, which adapts the C OMPUTE H EURISTIC function used to evaluate 
Revised: Algorithm 3 shows pseudo-code for the approach, which adapts the C OMPUTE H EURISTIC function used to evaluate a state.

Original: Algorithm 3 Online diversiﬁcation.

Original: Simultaneously diversify a set of orders Ω for SCP and compute the maximum over all induced SCP heuristic values for a given state s .

Original: 1: function C OMPUTE H EURISTIC ( Ω , s ) 2: if S ELECT ( s ) and time spent in function < T then 3: ω ← greedy order for s 4: if h SCP ω ( s ) > sup ω

Original: ∈ Ω h SCP ω ( s ) then 5: Ω ← Ω ∪ { ω } 6: return

Original: max ω ∈ Ω h SCP ω ( s ) a state.

Original: Before C OMPUTE H EURISTIC is called for the ﬁrst time, we initialize the set of heuristic orders Ω for SCP to be the empty set.
Revised: Before C OMPUTE H EURISTIC is called for the ﬁrst time, we initialize the set of heuristic orders Ω for SCP to be the empty set.

Original: 1 When evaluating a state s , we let the state selection function S ELECT decide whether to use s for diversifying Ω (line 2).
Revised: 1 When evaluating a state s , we let the state selection function S ELECT decide whether to use s for diversifying Ω (line 2).

Original: We discuss several state selection functions below, but all of them select the initial state for diversiﬁcation.
Revised: We discuss several state selection functions below, but all of them select the initial state for diversiﬁcation.

Original: If s is selected, we compute a greedy order ω for s (line 3) and check whether ω induces an SCP heuristic h SCP ω with a higher estimate for s than all previously stored orders (line 4).
Revised: If s is selected, we compute a greedy order ω for s (line 3) and check whether ω induces an SCP heuristic h SCP ω with a higher estimate for s than all previously stored orders (line 4).

Original: If that is the case, we store ω (line 5).
Revised: If that is the case, we store ω (line 5).

Original: Finally, we return the maximum heuristic value for s over all SCP heuristics induced by the stored orders (line 6).
Revised: Finally, we return the maximum heuristic value for s over all SCP heuristics induced by the stored orders (line 6).

Original: Compared to ofﬂine diversiﬁcation, this  algorithm has the advantage that it allows the A ∗ search to start immediately and it doesn’t need to sample states with random walks, but can judge the utility of storing an order based on states that are actually evaluated during the search. 
Revised: Compared to ofﬂine diversiﬁcation, this online diversiﬁcation algorithm has the advantage that it allows the A ∗ search to start immediately and it doesn’t need to sample states with random walks, but can judge the utility of storing an order based on states that are actually evaluated during the search. Compared to computing a saturated cost partitioning heuristic for each evaluated state (online-nodiv), online diversiﬁcation evaluates states much faster and consequently solves many more tasks.

Original: Time Limit
Revised: Time Limit

Original: For abstraction heuristics, the ofﬂine diversiﬁcation can perform two rather subtle optimizations compared to the online diversiﬁcation: after precomputing all SCP heuristics, we can delete all abstract transition systems from memory, since during the search we only need the abstraction functions, which map from concrete to abstract states.
Revised: For abstraction heuristics, the ofﬂine diversiﬁcation can perform two rather subtle optimizations compared to the online diversiﬁcation: after precomputing all SCP heuristics, we can delete all abstract transition systems from memory, since during the search we only need the abstraction functions, which map from concrete to abstract states.

Original: Furthermore, for abstractions that never contribute any heuristic information under the set of precomputed orders, we can even delete the corresponding abstraction functions (Seipp 2018).
Revised: Furthermore, for abstractions that never contribute any heuristic information under the set of precomputed orders, we can even delete the corresponding abstraction functions (Seipp 2018).

Original: While both optimizations often greatly reduce the memory footprint, the latter also speeds up the heuristic evaluation since we need to map the concrete state to its abstract counterpart for fewer abstractions.
Revised: While both optimizations often greatly reduce the memory footprint, the latter also speeds up the heuristic evaluation since we need to map the concrete state to its abstract counterpart for fewer abstractions.

Original: To allow the online diversiﬁcation to do these two optimizations, we need to stop the diversiﬁcation eventually.
Revised: To allow the online diversiﬁcation to do these two optimizations, we need to stop the diversiﬁcation eventually.

Original: We therefore introduce a time limit T and only select a state for diversiﬁcation (line 2) if the total time spent in C OMPUTEH EURISTIC is less than T .
Revised: We therefore introduce a time limit T and only select a state for diversiﬁcation (line 2) if the total time spent in C OMPUTEH EURISTIC is less than T .

Original: State Selection Strategies
Revised: State Selection Strategies

Original: We now discuss two possible instantiations of the S ELECT function, i.e., strategies for choosing the states for which to diversify the set of orders.
Revised: We now discuss three instantiations of the S ELECT function, i.e., strategies for choosing the states for which to diversify the set of orders.

Original: However, preliminary experiments showed that this only has a mild advantage over pure ofﬂine and pure online variants, so we only consider the pure variants here.

Original: Intervals
Revised: Intervals

Original: The ﬁrst strategy selects every i -th evaluated state for a given value of i .
Revised: The ﬁrst strategy selects every i -th evaluated state for a given value of i .

Original: The motivation for this strategy is to distribute the time for diversiﬁcation across the state space, in order to select states for diversiﬁcation that are different enough from each other to let the corresponding SCP heuristics generalize to many unseen states.
Revised: The motivation for this strategy is to distribute the time for diversiﬁcation across the state space, in order to select states for diversiﬁcation that are different enough from each other to let the corresponding SCP heuristics generalize to many unseen states.

Original: Note that for i =

Revised: Note that for i =

Original: this strategy selects all states until hitting the diversiﬁcation time limit T .
Revised: this strategy selects all states until hitting the diversiﬁcation time limit T .

Original: For i =1 and T = ∞ the resulting heuristic dominates the online SCP variant without diversiﬁcation (online-nodiv), because both heuristics compute the same SCP heuristic for the currently evaluated state, but the variant with diversiﬁcation also considers all previously stored orders.
Revised: For i =1 and T = ∞ the resulting heuristic dominates the online SCP variant without diversiﬁcation (online-nodiv), because both heuristics compute the same SCP heuristic for the currently evaluated state, but the variant with diversiﬁcation also considers all previously stored orders.

Original: Novelty
Revised: Novelty

Original: This strategy makes the notion of “different states” explicit by building on the concept of novelty (Lipovetzky and Geffner 2012).
Revised: This strategy makes the notion of “different states” explicit by building on the concept of novelty (Lipovetzky and Geffner 2012).

Original: Novelty is deﬁned for factored states spaces, i.e., where each state s is deﬁned by a set of atoms (atomic propositions) that hold in s .
Revised: Novelty is deﬁned for factored states spaces, i.e., where each state s is deﬁned by a set of atoms (atomic propositions) that hold in s .

Original: The novelty of a state s is the size of the smallest conjunction of atoms that is true in s and false in all states previously evaluated by the search.
Revised: The novelty of a state s is the size of the smallest conjunction of atoms that is true in s and false in all states previously evaluated by the search.

Original: For a given value of k , the novelty strategy selects a state if it has a novelty of at most k .
Revised: For a given value of k , the novelty strategy selects a state if it has a novelty of at most k .

Revised: Bellman The last strategy selects a state s if the maximum over the currently stored SCP heuristics h SCP Ω violates the

Revised: Bellman optimality equation (1957) for s and its successor states, i.e., if h SCP Ω ( cost , s ) < min s a −→ s (cid:48)

Revised: ∈ T h SCP Ω ( cost , s (cid:48) )

Revised: + cost ( a ) , where T is the set of transitions in the planning task.

Revised: Whenever the Bellman optimality equation is violated for a state s , we know that the current estimate for s is lower than the true goal distance of s , in which case it seems prudent to select s for diversiﬁcation.

Original: Experiments
Revised: Experiments

Original: We implemented online diversiﬁcation for saturated cost partitioning in the Fast Downward planning system (Helmert 2006) and used the Downward Lab toolkit (Seipp et al. 2017) for running experiments.
Revised: We implemented online diversiﬁcation for saturated cost partitioning in the Fast Downward planning system (Helmert 2006) and used the Downward Lab toolkit (Seipp et al. 2017) for running experiments on Intel Xeon Silver 4114 processors.

Original: Our benchmark set consists of all 1827 tasks without conditional effects from the optimal sequential tracks of the International Planning Competitions 1998–2018.
Revised: Our benchmark set consists of alltasks without conditional effects from the optimal sequential tracks of the International Planning Competitions 1998– 2018.

Original: We limit time by 30 minutes and memory by 3.5 GiB. All benchmarks, code and experiment data have been published online.
Revised: We limit time by 30 minutes and memory by 3.5 GiB. All benchmarks, code and experiment data have been published online.

Original: For the heuristic set on which SCP operates, we use the combination of pattern databases found by hill climbing (Haslum et al. 2007), systematic pattern databases of sizesand 2 (Pommerening, R¨oger, and Helmert 2013) and Cartesian abstractions of landmark and goal task decompositions (Seipp and Helmert 2018). 
Revised: For the heuristic set on which SCP operates, we use the combination of pattern databases found by hill climbing (Haslum et al. 2007), systematic pattern databases of sizesand 2 (Pommerening, R¨oger, and Helmert 2013) and Cartesian abstractions of landmark and goal task decompositions (Seipp and Helmert 2018). When comparing planning algorithms, we focus on the number of solved tasks, i.e., the coverage of a planner and its time score (used for the agile track of IPC 2018).

Revised: The time score of a planner P for a task that

Revised: P solves in t seconds is deﬁned as 1 − log( t ) log( T ) , where T is the time limit, i.e., 1800 seconds in our case.

Revised: The time score is 0 if P fails to solve the task within 1800 seconds.

Revised: The total coverage and time score of a planner is the sum of its scores over all tasks.

Original: State Selection Strategies
Revised: Reevaluating States

Revised: The ofﬂine diversiﬁcation algorithm ﬁnds a set of heuristic orders and maximizes over the corresponding SCP heuristics during the search. With such a ﬁxed set of orders, the overall heuristic value of a state never changes. When we diversify the set of orders online during the search however, the heuristic estimate of a state s can increase after the time when s is generated, evaluated and added to the open list.

Revised: Consequently, the heuristic estimates of states in the open list may be too low and it might be beneﬁcial to reevaluate each state when retrieving it from the open list and postponing its expansion if its heuristic value has increased.

Revised: To test this, we compare online diversiﬁcation without and with state reevaluations in Table 1 (columns on-stable and online ).

Revised: The results show that reevaluating states increases the number of solved tasks in three domains (scanalyzer, tetris and tidybot) and never lets coverage decrease.

Revised: Our implementation uses the fact that we only need to reevaluate a state for the additional orders that we stored since its last evaluation.

Revised: This minimizes the overhead incurred by the state reevaluations and makes the online variant solve tasks faster than on-stable in many domains (see right part of Table 1).

Revised: Due to these results we let all online diversiﬁcation variants reevaluate states in the experiments below.

Revised: State Selection Strategies

Original: In the ﬁrst experiment, we compare the different instantiations of the S ELECT function.
Revised: In the next experiment, we compare the different instantiations of the S ELECT function.

Original: The left and middle parts of Table 1 hold per-domain and overall coverage results for the interval strategy with different intervals and the novelty strategy for novelty 1 and 2.
Revised: The left and middle parts of Table 2 hold per-domain and overall coverage results for the interval strategy with different intervals, the novelty strategy for k =1 and k =2 and the Bellman strategy. All strategies use [Table]

Original: All strategies use a time limit of 1000 seconds for the online diversiﬁcation.
Revised: nod [Table]

Original: We see that the overall coverage is roughly the same for the interval variants (1141–1143 solved tasks) and that the higher intervals 100 and 1000 have an edge over the lower intervals 1 andin per-domain coverage comparisons.
Revised: [Table] movie [Table] [Table]

Original: The novelty variants are preferable to all interval variants in per-domain comparisons and also solve more tasks in total (1149 andtasks, respectively).
Revised: mprime [Table] [Table] mystery (

Original: These results suggest that explicitly focusing on states with previously unseen atom combinations
Revised: [Table]

Original: T =1000s i n t e r v a li n t e r v a li n t e r v a li n t e r v a lnov e lt ynov e lt yT =1000s T = ∞
Revised: [Table] [Table] nomystery

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: openstacks

Original: [Table] 
Revised: [Table] parcprinter

Original: [Table] 
Revised: [Table] parking

Original: [Table]
Revised: [Table]

Revised: petri

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: psr

Original: [Table]
Revised: [Table]

Original: i n
Revised: rovers

Original: [Table]
Revised: [Table]

Original: s on li n e - 1 s on li n es o f ﬂ i n e10 s o f ﬂ i n es on li n e100 s o f ﬂ i n e100 s on li n es
Revised: satellite (36) [Table] [Table] scanalyzer

Original: C ov [Table]
Revised: [Table] [Table]

Original:  r  a 
Revised: stable and online use interval selection with i = 10K. b e ll m a n nov e lt yi n t e r v a li n t e r v a lnov e lt yi n t e r v a lK i n t e r v a lK i n t e r v a li n t e r v a l10 K T = s T = ∞

Original: [Table]
Revised: [Table]

Original: e
Revised: a time limit for

Original: [Table]
Revised: We see that overall coverage is similar for all interval and novelty variants (1153–1159 solved tasks) and that the Bellman strategy solves fewer tasks in total than the other strategies.

Original: helps to ﬁnd stronger SCP heuristics and therefore we use the novelty-2 strategy in the experiments below.
Revised: We obtain the highest total coverage by selecting every ten thousandth evaluated state ( interval-10K ) and therefore we use this strategy in all other experiments.

Original: Time Limit
Revised: Time Limit

Original: The middle and right parts of Table 1 conﬁrm that we need a time limit for the online diversiﬁcation.
Revised: The middle and right parts of Table 2 conﬁrm that we need a time limit for the online diversiﬁcation.

Original: For all state selection strategies total coverage decreases when the time limit of 1000 seconds for the diversiﬁcation is lifted. 
Revised: For all state selection strategies total coverage decreases when the time limit of 1000 seconds for the diversiﬁcation is lifted. The coverage loss is higher, the more states we may select for diversiﬁcation.

Original: The coverage of the novelty-1 version only decreases by 11 tasks, because the number ofstates selected for diversiﬁcation is limited by the number of atoms A in the planning task. 
Revised: For example, the coverage of the novelty-1 variant only decreases by 28 tasks, because the number of selected states is limited by the number of atoms A in the planning task. For novelty-2 coverage decreases by 99 tasks, because at most | A | 2 states can be selected. Samples

Original: For novelty-2 this bound is | A | 2 and  the interval versions are only bounded by the number of evaluated states, which is why a time limit is more important for these variants.
Revised: The ofﬂine and online diversiﬁcation algorithms differ in two main respects: for which states they compute orders and which states they use to decide whether to store an order. For both of these decisions, the ofﬂine variant uses sample states obtained with random walks, whereas the online variant uses states evaluated during the search. In this subsection, we analyze whether online diversiﬁcation beneﬁts from considering randomly sampled states.

Original: [Figure] [Figure]
Revised: s a m p l e s [Table]

Original: s o l v e d t a s k s online-1000s ofﬂine-100s online-100s ofﬂine-1000s online-nodiv
Revised: They differ in they obtained with When using the state method ˆ S only contains the currently evaluated state (as in Algorithm 3). The both method sets ˆ S to the union of the samples and the evaluated state. For an explanation of the data, see Table 2.

Original: Ofﬂine vs. Online Diversiﬁcation
Revised: [Table] The [Table] variants use the interval

Revised: [Table]

Revised: 10K

Revised: [Table]

Revised: It is unlikely that computing orders for randomly sampled states is preferable to computing orders for states that the search actually evaluates.

Revised: However, it could be beneﬁcial to use a set of sample states when judging whether an order should be stored.

Revised: In Table 3, we evaluate this hypothesis by comparing three different choices for the question which states to consider when deciding whether an order should be stored.

Revised: The data shows that we solve almost the same number of tasks in total and per domain regardless of whether we take into account only a single state, a set of 1000 samples or both.

Original: Having established that we need a time limit foronline diversiﬁcation and that the novelty-2 strategy leads to the highest coverage, we  now evaluate different time limits and compare the resulting algorithms to their ofﬂine counterparts.
Revised: However, storing an order when it improves the heuristic value for the currently evaluated state results in shorter runtimes than for the other two variants in Table 3, which is why we only consider this variant in Algorithmand all other experiments. Ofﬂine vs. Online Diversiﬁcation We now evaluate different time limits and compare the resulting algorithms to their ofﬂine counterparts.

Original: Tableconﬁrms the result from Seipp, Keller, and Helmert (2020) that we cannot simply reduce the time for ofﬂine diversiﬁcation (to 1 or 10 seconds) in order to reduce overall runtime, without sacriﬁcing total coverage. 
Revised: The top part of Table 4 conﬁrms the result from Seipp, Keller, and Helmert (2020) that we cannot simply reduce the time for ofﬂine diversiﬁcation (to 1 or 10 seconds) in order to minimize overall runtime, without sacriﬁcing total coverage. Ofﬂine diversiﬁcation solves the highest number of tasks (1159) with a time limit T of 100 seconds and slightly fewer tasks (1156) with T =1000s. Using lower or higher time limits leads to solving much fewer tasks.

Revised: The results are similar for

Original: For a time limit ofseconds, ofﬂine diversiﬁcation solves more tasks than online diversiﬁcation in 6 domains, while the opposite is true indomains.
Revised: [Figure] failed [Figure] online diversiﬁcation, which solves the most tasks (1159) for

Original: The picture is reversed for a time limit of 1000 seconds: here, online diversiﬁcation has an edge over the ofﬂine variant in a per-domain comparison (7 vs. 3 domains).
Revised: T =1000s and slightly fewer tasks (1153–1154) for T =100s and T =1200s. Online diversiﬁcation is less susceptible to the chosen time limit than ofﬂine diversiﬁcation: while the difference between the maximum and minimum coverage score for ofﬂine diversiﬁcation is 103 tasks, the corresponding value for online diversiﬁcation is only 57 tasks. Tableshows detailed coverage and time score results for the ofﬂine and online variants that use at most 1000 seconds for diversiﬁcation (among two other variants). Before we analyze the runtimes of the different variants, we compare the number of orders stored by ofﬂine and online diversiﬁcation (using 1000 seconds for diversiﬁcation) in Figure 1.

Revised: We can see that the online variant tends to store fewer orders than the ofﬂine counterpart, often by more than one order of magnitude.

Original: Online diversiﬁcation for 1000 seconds also leads to the highest total coverage of 1151 tasks, solving 5 tasks more than the best previously-known method for ﬁnding SCP heuristics.
Revised: More precisely, online diversiﬁcation stores fewer orders than ofﬂine diversiﬁcation for 1221 tasks, while the opposite is the case for 296 tasks. For SCP the increased accuracy from using more orders usually outweighs the increased evaluation time (Seipp, Keller, and Helmert 2020).

Revised: Therefore, Figure 1 suggests that the online diversiﬁcation stores fewer redundant orders than the ofﬂine diversiﬁcation, because otherwise the coverage gap between the two variants (3 tasks) would be larger.

Original: Not only does online diversiﬁcation  obtain high coverage scores, but it also drastically reduces the overall runtime for many tasks compared to ofﬂine diversiﬁcation. 
Revised: Not only does online diversiﬁcation select useful orders and obtain high coverage scores, but it also drastically reduces the overall runtime for many tasks compared to ofﬂine diversiﬁcation. The bottom part of Table 4 reveals that the time score of all online variants is higher than the best time score of all ofﬂine variants. The time score gap between the two variants is 129.4 points for T = 1s

Original: Figureshows the cumulative number of solved tasks over time by the strongest algorithms from Table 2 and the variant that computes an SCP heuristic for each evaluated state without  any diversiﬁcation.
Revised: and it grows to 880. points for T =1500s. Figure 2 shows the cumulative number of solved tasks over time by ofﬂine and online diversiﬁcation (with T =1000s) and the variant that computes an SCP heuristic for each evaluated state without storing any orders.

Original: The latter variant (online-nodiv) solves the simpler tasks quickly, but only reaches a total coverage of 760 tasks.
Revised: The lat- [Figure] [Figure] s o l v e d t a s k s online-1000s ofﬂine-1000s online-nodiv ter variant (online-nodiv) solves the simpler tasks quickly, but only reaches a total coverage of 766 tasks.

Original: The ofﬂine variants that diversify for 100 and 1000 seconds achieve a much higher total coverage ( and 1143 tasks, respectively), but they can only start ﬁnding solutions after their diversiﬁcation phase ended.
Revised: The ofﬂine variant achieves a much higher total coverage (1156 tasks), but it can only start ﬁnding solutions after its diversiﬁcation phase ended.

Original: The online variants with diversiﬁcation combine the advantages of the other  approaches and achieve both short runtimes and high total coverage.
Revised: The online variant with diversiﬁcation combines the advantages of the other two approaches and achieves both short runtimes and high total coverage (1159 tasks).

Original: For example, online-1000s solves 999 tasks before ofﬂine-1000s even ﬁnishes the diversiﬁcation phase.
Revised: For example, online-1000s solves 1121 tasks before ofﬂine-1000s even ﬁnishes the diversiﬁcation phase.

Original: After reaching the diversiﬁcation time limit, the online and ofﬂine variants solve roughly the same number of additional tasks per time step.
Revised: After reaching the diversiﬁcation time limit, the online and ofﬂine variants solve roughly the same number of additional tasks per time step.

Original: The plot also shows that it is useful to reduce the diversiﬁcation time limit from 1000 to 100 seconds if fast overall runtimes are more important than maximum coverage after 30 minutes.
Revised: For all time limits between 1 and 1800 seconds, online diversiﬁcation solves more tasks than ofﬂine diversiﬁcation and the online-nodiv variant. The right part of Table 1 holds per-domain time scores for the algorithms in Figure 2.

Original: This holds for both the ofﬂine and online variants. For all of the four tested time limits (1s and 10s not depicted) and almost all overall time limits, however, online diversiﬁcation solves more tasks than ofﬂine diversiﬁcation.
Revised: The numbers show that online diversiﬁcation is faster than ofﬂine diversiﬁcation in all domains, and usually achieves much higher time scores (columns ofﬂine and online in Table 1).

Original: Related Work
Revised: Related Work

Original: The work that is most closely related to ours simultaneously reﬁnes a set of Cartesian abstraction heuristics and a set of SCP heuristics over them during an A ∗ search (Eiﬂer and Fickert 2018).
Revised: The work that is most closely related to ours simultaneously reﬁnes a set of Cartesian abstraction heuristics and a set of SCP heuristics over them during an A ∗ search (Eiﬂer and Fickert 2018).

Original: Whenever the maximum over the SCP heuristics violates the Bellman optimality equation (1957) for a state s and its successor states, the authors either reﬁne one of the abstractions until the heuristic estimate for s increases, merge two abstractions or compute a new greedy order ω for s (using the h scoring function, Seipp, Keller, and Helmert 2020) and add h SCP ω to the set of SCP heuristics.
Revised: Whenever the maximum over the SCP heuristics violates the Bellman optimality equation (1957) for a state s and its successor states, the authors either reﬁne one of the abstractions until the heuristic estimate for s increases, merge two abstractions or compute a new greedy order ω for s (using the h scoring function, Seipp, Keller, and Helmert 2020) and add h SCP ω to the set of SCP heuristics.

Original: Their strongest algorithm compares favorably against a version that only reﬁnes the abstractions ofﬂine and only computes a single SCP heuristic over them.
Revised: Their strongest algorithm compares favorably against a version that only reﬁnes the abstractions ofﬂine and only computes a single SCP heuristic over them.

Original: However, both the online and the ofﬂine version are outperformed by the version that diversiﬁes a set of SCP heuristics over a ﬁxed set of Cartesian abstraction heuristics, i.e., the ofﬂine SCP variant we describe in Algorithm 2.
Revised: However, both the online and the ofﬂine version are outperformed by the version that diversiﬁes a set of SCP heuristics over a ﬁxed set of Cartesian abstraction heuristics, i.e., the ofﬂine SCP variant we describe in Algorithm 2.

Original: The literature contains additional appproaches that improve heuristics online during the search.
Revised: The literature contains additional approaches that improve heuristics online during the search.

Original: For example, the
Revised: For example, the

Original: SymBA ∗ planner repeatedly switches between a symbolic forward search and symbolic backward searches in one of multiple abstractions (Torralba, Linares L´opez, and Borrajo 2016).
Revised: SymBA ∗ planner repeatedly switches between a symbolic forward search and symbolic backward searches in one of multiple abstractions (Torralba, Linares L´opez, and Borrajo 2016).

Original: In the setting of satisiﬁcing planning, Fickert and Hoffmann (2017) reﬁne the FF heuristic (Hoffmann and
Revised: In the setting of satisﬁcing planning, Fickert and Hoffmann (2017) reﬁne the FF heuristic (Hoffmann and

Original: Nebel 2001) during enforced hill-climbing and greedy bestﬁrst searches.
Revised: Nebel 2001) during enforced hill-climbing and greedy bestﬁrst searches.

Original: As a ﬁnal example, Franco and Torralba (2019) interleave the precomputation of a symbolic abstraction heuristic and the symbolic search that uses it, by iteratively switching between the two phases.
Revised: Asaﬁnalexample, Franco and Torralba (2019) interleave the precomputation of a symbolic abstraction heuristic and the symbolic search that uses it, by iteratively switching between the two phases.

Original: In each round they double the amount of time given to each phase.
Revised: In each round they double the amount of time given to each phase.

Original: Our work is orthogonal to theirs since the two approaches focus on interleaving two different types of precomputation with the search.
Revised: Our work is orthogonal to theirs since the two approaches focus on interleaving two different types of precomputation with the search.

Original: Conclusions
Revised: Conclusions

Original: The best previously-known method for computing diverse
Revised: The best previously-known method for computing diverse

Original: SCP heuristics uses a ﬁxed amount of time for sampling states and computing SCP heuristics for them.
Revised: SCP heuristics uses a ﬁxed amount of time for sampling states and computing SCP heuristics for them.

Original: It yields strong heuristics, but needs a long precomputation phase.
Revised: It yields strong heuristics, but needs a long precomputation phase.

Revised: Computing an SCP heuristic for each evaluated state yields even better estimates and needs no precomputation phase, but it greatly slows down the search.

Revised: We showed that by diversifying SCP heuristics online , we can combine the strengths of both approaches and obtain an algorithm that needs no sample states nor precomputation phase, evaluates states quickly and achieves high coverage.

Revised: Currently, the strongest optimal classical planners compute multiple cost partitionings over abstraction heuristics and use them in an A ∗ search.

Original: We showed that by performing the diversiﬁcation online during the search  we drastically speed up the overall algorithm and even solve slightly more tasks.
Revised: There are three steps that can take long before these planners can start their search: deciding which abstractions to build (i.e., pattern selection for pattern database heuristics), building the abstractions and computing orders for cost partitioning algorithms. Franco and Torralba (2019) show how to interleave the search with building an abstraction and our paper shows how to efﬁciently compute orders online. It will be interesting to see how we can decide during the search which abstractions to build and how we can combine all of these techniques. Acknowledgments

Original: Future work could evaluate whether it is beneﬁcial to take not only the currently evaluated state, but other states from the open list, into account when judging whether a new order for SCP should be stored.
Revised: We thank Malte Helmert and the anonymous reviewers for their insightful comments. We have received funding for this work from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 817639). Online Reﬁnement

Revised: https://doi.org/10.5281/zenodo.

Revised: doi

Revised: 10.5281/zenodo.790461.

Revised: URL https://doi.org/ 10.5281/zenodo.