Learning To Estimate Search Progress Using Sequence Of States
Abstract
Many problems of interest can be solved using heuristic search algorithms.
When solving a heuristic search problem, we are often interested in estimating search progress, that is, how much longer until we have a solution.
Previous work on search progress estimation derived formulas based on some relevant features that can be observed from the behavior of the search algorithm.
In this paper, rather than manually de- riving such formulas we leverage machine learning to automatically learn more accurate search progress predictors.
We train a Long Short-Term Memory (LSTM) network, which takes as input sequences of states expanded by the search al- gorithm, and predicts how far along the search we are.
Importantly, our approach still treats the search algorithm as a black box, and does not look into the contents of search states.
An empirical evaluation shows our technique outperforms previ- ous search progress estimation techniques.
Introduction
Many interesting and challenging real-world problems can be deﬁned as search problems, where the objective is to ﬁnd a path in a state space (which is often implicitly deﬁned by a black-box successor generator) from some given initial state to a goal state.
Often, we can use a heuristic function which estimates the distance from any given state to the goal to better guide this search.
Many heuristic search algorithms have been developed to solve such problems, the most well-known of which are probably A ∗ (Hart, Nilsson, and Raphael 1968) and Greedy best-ﬁrst search (GBFS) (Doran and Michie 1966).
Search algorithms can have unpredictable behavior, as we typically do not know the topology of the search space (if we did, perhaps we would not need to use a search algorithm).
While there are some results when the search problem is described symbolically (Hoffmann 2011), these results do not apply for black-box search problem.
Other work has addressed estimating search effort in terms of number of expanded nodes (Lelis, Stern, and Sturtevant 2014;
Belov et al. 2017) or the cost of an optimal solution (Lelis et al.
– we discuss these and others in more detail later.
Importantly, these approaches are all ofﬂine – they give their predictions before the search starts.
We are interested in predicting remaining search effort online , that is, in estimating how much progress the search algorithm has already made.
This problem was ﬁrst introduced by Thayer, Stern, and Lelis (Thayer, Stern, and Lelis 2012), who also derived some formulas for estimating search progress based on relatively simple search features, which we explain in more detail later.
Search progress estimation is a useful tool not just for letting a user know whether they have time to make a cup of coffee before search completes, but also in making decisions about the search.
For example, remaining search time estimates are used in situated temporal planning (Cashmore et al. 2018) to prioritize search nodes which have a higher probability to lead to a timely solution.
Such estimates can also be used to decide when to perform a restart in a Branch-and-Bound search (Anderson et al. 2019) for solving a Mixed Integer Program.
In this paper, we leverage recent developments in machine learning to automatically learn more complex search predictors based on a given training set.
Speciﬁcally, we view the search algorithm as a black-box that emits a sequence of states (we also treat the node which the search algorithm expands as a black box), and train a Long Short-Term Memory (LSTM) network (Hochreiter 1998) to predict remaining search effort.
We conducted an empirical evaluation on a large set of planning domains, comparing our technique to previous work (Thayer, Stern, and Lelis 2012).
The results indicate that our technique achieves state-of-the-art performance in search progress estimation.
Background
In this section we review some of the necessary background on LSTM networks and heuristic search.
LSTM Networks
Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber 1997) are a speciﬁc type of Recurrent
Neural Network (Medsker and Jain 2001) – a deep learning approach which is especially suitable for learning from sequences.
LSTMs were ﬁrst introduced in order to solve the vanishing gradient problem in conventional Recurrent
Neural Networks (Hochreiter 1998), and have been extensively used in natural language processing, e.g., (Sutskever,
Vinyals, and Le 2014; Gers, Schmidhuber, and Cummins 2000).
We now brieﬂy review how LSTM networks are built, but refer the interested reader to a detailed survey (Yu et al.
for more details.
An LSTM network consists of several LSTM cells, arranged in sequence.
Each LSTM cell has a hidden state, and 3 gates: input, forget and output, that manage the reading, writing and memory update, respectively.
The cell learns gate weights and uses them to change the input and the hidden state based on the previous input in the sequence.
The cell processes input x t together with the hidden state h t − 1 (which is the result of processing the previous input) by ﬁrst passing them to the forget gate, which has a Sigmoid activation function.
The output of the forget gate is a number between 0 and 1 for each element in the cell state C t – the information we are passing to the next LSTM cell.
Next, the processing continues through the input gate, which has 2 activation functions, Sigmoid and tanh.
The input is the same as the forget layer x t and h t − 1 .
The Sigmoid activation decides which values to update and the tanh activation creates the values themselves.
The outputs from both of these are multiplied.
Finally, processing continues to the output gate that gets x t and h t − 1 as inputs and goes through a Sigmoid activation which decides what parts will be output.
The output also goes through a tanh activation, which is multiplied by the output of the Sigmoid activation.
This is all captured by the following equations, which describe the operation of each LSTM cell:
• forget gate: f t = σ ( W f · [ h t − 1 , x t ] + b f ) • input gate: i t
= σ ( W i · [ h t − 1 , x t ] + b i ) • new cell state value: (cid:102) C T =
tanh
( W c · [ h t − 1 , x t ] + b C ) • update cell state: C T =
f t ∗ C t − 1 + i t ∗ (cid:102) C T • output gate: o t
= σ ( W 0 [ h t − 1 , x t ] + b 0 ) • hidden state: h t =
o t ∗ tanh ( C t )
Heuristic search
As previously mentioned, many problems of interest can be modeled as state-space search problems (Bonet and Geffner 2001).
A state space is formally represented as a tuple T =
(cid:104) S, S 0 , S G , A, f, c (cid:105) , where:
• S is a ﬁnite and non-empty set of states, • S 0 ∈ S is the initial state,
• S G ⊆ S is a non-empty set of goal states, • A ( s ) ⊆
A denotes the actions applicable in each state s ∈ S , • f : S ×
A → S is the state transition function, such that applying action a in state s leads to state f
( s, a ) , and • c ( s, a ) is the cost of performing action a in state s
Uninformed search algorithms such as Breadth-First
Search (BFS) can be used to solve small state-space search problems.
However, to solve larger problems, we typically also use a heuristic which estimates the distance from any given node to the goal.
Formally, a heuristic is a function mapping the states S to some real number (or inﬁnity to indicate a dead-end), that is h : S → R + ∪ {∞} .
Most search algorithms deﬁne some priority function f ( s ) over the states.
Typically, f ( s ) is obtained from h ( s ) , the heuristic estimate of the distance to go from s , and g ( s ) , the cost of the best known path to s so far.
For example, A ∗ (Hart, Nilsson, and Raphael 1968) expands the node with the lowest f A ∗ ( s ) =
g ( s ) + h ( s ) , while GBFS expands the node with the lowest f GBFS ( s )
=
h ( s ) .
We will use both f, g and h depend on the search algorithm as features for each search node.
Search progress estimation
We can now formally describe the problem we address in this paper – search progress estimation.
Our deﬁnition here is adapted from previous work (Thayer, Stern, and Lelis 2012).
Let A be a search algorithm, and P = (cid:104) T, h (cid:105) be a heuristic search problem, which consists of a search space
T and a heuristic function h .
Denote by E A ( P ) the number of nodes expanded by search algorithm A while attempting to solve a search problem P .
Let Rem A
( P, E A ( P )) be the number of remaining nodes that are going to be expanded by A when solving P after A has already expanded E A ( P ) nodes, that is, Rem A ( P, E A ( P )) =
E A ( P ) − E A ( P ) .
Deﬁnition 1 (Search Progress) .
The search progress of algorithm A solving problem P after expending E A ( P ) nodes is: Prog
[Equation]
As Deﬁnition 1 shows, search progress is a number between 0 and 1, which indicates what fraction of the total search effort to solve the problem we have already expended.
For example if we expanded node number 1,000 out of 10,000 nodes that are going to be expanded in total, the search progress will be 1 , 000 / 10 , 000 = 0 .
This allows comparing search progress between problems of different sizes on the same scale.
The metric we use to evaluate the accuracy of search progress estimation (following (Thayer,
[-• h 
 = h ( S 0 ) – the heuristic value of the initial state. This value stays constant throughout the search, but-]Stern, and Lelis 2012)) is Mean Absolute Error (MAE):
For each node we compute the absolute error between the predicted progress and the real progress.
The accuracy of a predictor on a given problem is the mean of these absolute errors over all the states the search expanded.
Related Work
Having described the necessary background, we can now con review related work in this ﬁeld.
Some previous work attempted to predict search effort in speciﬁc domains, e.g., predicting the number of nodes expanded by A ∗ for the 15puzzle (Breyer and Korf 2008).
Other work used sampling to estimate search effort (Lelis et al. 2014;
Belov et al. 2017;
Hutter et al. 2014) or the cost of a solution (Lelis, Stern, and Sturtevant 2014) in a more general setting.
However, as previously mentioned, these are all ofﬂine methods, which attempt to make a prediction before the search starts.
We now review in more detail the only prior work we are aware of that addresses online search progress estimation (Thayer, Stern, and Lelis 2012).
Several search progress estimators were proposed, which we now describe, and later use as baselines in our empirical evaluation:
Velocity-Based Search Speed Estimator (VeSP)
The
Velocity-Based Search Speed Estimator (VeSP) calculates the velocity during the search when E A ( P ) denoted as the number of expanded nodes by the average velocity as:
[Equation]
It then uses the velocity V and the current h min to estimate the remaining search effort (number of expanded nodes)
[Equation]
Finally, it uses this estimate to predict search progress by using:
[Equation]
Vacillation-Based Search Speed Estimator (VaSP)
The
Vacillation-Based Search Speed Estimator (VaSP) differs from VeSP by using node serial numbers to estimate the expansion delay (Dionne, Thayer, and Ruml 2011), that is, the average number of expansions between when a node is generated and when it is expanded, denoted ∆ e .
The expansion delay is combined with h min to predict remaining search effort as:
[Equation]
As in VeSP, the estimate of search progress is:
[Equation]
Path-Based Progress Estimator (PBP)
We ﬁrst describe the Naive PBP (NPBP), which estimates search progress at a given node n by looking at the ratio between the cost until the current node ( g ( n ) ) and the total estimated cost ( g ( n ) + h ( n
[Equation]
The main problem with NPBP is that it depends on the current node, but in many cases there are other open nodes with higher progress.
Thus the PBP estimator returns the maximum NPBP estimate among the nodes expanded far.
Distribution-Based Progress Estimator (DBP)
The
Distribution-Based Progress Estimator (DBP) estimates the search progress using data observed during search.
It relies on counting how many nodes are expanded by the search for each value of d ( n ) – an estimate of remaining plan length .
We remark that in unit-cost domains d ( n ) =
h ( n ) , and where action costs are non-uniform, d ( n ) can usually be estimated alongside h ( n ) .
Deﬁne c
[ d i ] as the count of how many nodes had value d ( n ) =
d i during search.
DBP ﬁts a second degree polynomial ˆ c to c , in order to estimate progress.
The DBP estimate is then deﬁned as:
[Equation]
where m is the highest d -value with a count higher than 0.
Learning to Predict Search Progress
We now describe our technique for search progress estimation, which is based on supervised machine learning.
More precisely, given a training set of search nodes with their correct search progress, we attempt to predict search progress for new search nodes.
Input Features: We begin by deﬁning the input features to our predictor. First, note that any expansion-based forward search algorithm works by expanding nodes in sequence. Thus, we can view the search algorithm as a black-box that outputs a sequence of nodes to expand. This motivates us to use a Recurrent Neural Network (as we discuss next), looking at the last k nodes that were expanded.
However, for each of these nodes, it is also important to know something about where it came from – the path from the initial node to that node.
Thus, for each of these k nodes, we also look at their ancestors m layers up.
In this paper, we used k = 30 and m = 2 , resulting in collecting features from k · ( m +1) =
nodes, as illustrated in Figure 1.
For each of these k · ( m +1) nodes, we collect the following features:
• g ( n ) – the cost of the best known path to node n .
• h ( n ) – the heuristic estimate of the cost from n to the goal.
• f ( n ) – the f -value for n the search algorithm uses.
• b ( n ) – the branching factor of n , that is, the number of successor node n has.
• N ( n ) – the serial number
of n , that is, how many nodes were expanded before n .
We remark that this number was previously used to measure the expansion delay (Dionne,
Thayer, and Ruml 2011).
Additionally, we use some global features, which capture a snapshot of the current node of the search:
{+• h 
+}
= h ( S 0 ) – the heuristic value of the initial state.
This value stays constant throughout the search, but is important for estimating the cost of the solution that will be returned.
• h min – the minimal h
-value we have seen so far among the expanded nodes.
• Nh min – the number of nodes we expanded since the last time h min was updated.
• f max – the maximum f value we have seen so far.
These features are used in the previous work (Thayer,
Stern, and Lelis 2012), with the exception of Nh min , which is used to get some information about whether search seems to be stuck in a heuristic plateau (Asai and Fukunaga 2017).
A heuristic plateau is an area of the search space where the heuristic assign similar values to all states, and thus heuristic guidance is not very useful.
Search algorithms typically spend most of their time in heuristic plateaus, as otherwise, the heuristic would lead the search algorithm directly to the goal.
N is the node that was expanded last, N − 1 the node expanded before it, and so on.
Parent() and Grandparent() are obtained by following the parent pointers for the indicated node.
We record the value of these features at the time each node was expanded, so we have k values for each of these features.
These input features can be seen as a matrix of real numbers, of dimension 5 k ( m + 1) + 4 k , which are used as the input (features) to our network.
Note that for the ﬁrst k states, we use feature values of 0 for the undeﬁned nodes (i.e., nodes with a negative serial number).
Network Structure:
Our network 1 consists of 15 LSTM layers, followed by a fully connected layer which reduces the dimension by half (from 450 to 225), then a dropout layer with 50% dropout and Rectiﬁed Linear Units (ReLU) activations, and ﬁnally another fully connected layer which outputs our prediction – a single number.
The parameters here (number of LSTM layers, dimensions, dropout rate, etc.) were determined by manual tuning over preliminary data.
Automatically setting these parameters is future work.
Training:
Given a planning problem, we ﬁrst solve it using a planner running a given search algorithm and heuristic, and generate the true label (the correct search progress) for each node.
During training, we sample 1,000 nodes uniformly at random from each problem in our dataset, in order to avoid bias to larger problems.
We use the node serial number to obtain the true label of search progress – for node n the true label is N ( n ) /N
( g n ) , where g n is the goal node found by the search algorithm.
We used Adam as an optimiser with a learning rate of 0.001 and a batch size of 1024, and Mean Squared Error (MSE) as the loss function. .
Having described our technique and the previous work on search progress estimation, we can now compare them empirically.
We now describe this empirical evaluation.
In order to compare search progress estimates on various problems with different heuristics and different search algorithms, we chose to use planning benchmarks from past
International Planning Competitions, as this allows us to implement our technique only once.
Speciﬁcally, we extended
Pyperplan (Alkhazraji et al. 2020) under license GNU General Public License 3 (GPLv3) to output the features of the nodes it expands during search.
This data is then used for training and testing of search progress estimators, as we describe next.
Our dataset started with planning problems from 21 IPC domains – speciﬁcally, the benchmarks which are part of the Pyperplan repository.
These domains have a total ofplanning problem instances.
For each instance, we ran 4 conﬁgurations of Pyperplan, choosing one of two search algorithms (A* or GBFS) with one of two heuristics (HFF (Hoffmann and Nebel 2001) or Lm-Cut (Helmert and Domshlak 2009)) with a time limit of 24 hours and a memory limit of 1,000,000 expanded nodes.
From the solved instances we omitted the instances which were solved using less than 1,000 expanded nodes (see table 1 solved problems), as these are solved so quickly that search progress estimation is useless, and it would have made sampling 1,000 nodes from each instance difﬁcult.
Experimental setup
We compare our technique to the techniques introduced by previous work (Thayer, Stern, and Lelis 2012), which we described in detail above: VaSP, VeSP, PBP, and DBP.
For
VaSP, we used a moving average over the last 200 nodes to estimate expansion delay.
For DBP we used NumPy to ﬁt the polynomial to the data at hand.
All of our experiments were run on a server with 72 Intel
Xeon E5-2695 CPUs (utilizing at most 18 processes in parallel (Tange et al. 2011) (available under GNU GENERAL
PUBLIC LICENSE). The deep learning was implemented in PyTorch, and run on 2 Nvidia Tesla M60 GPUs.
On this hardware the inference takes 0.05ms per node.
As our technique is based on learning from a training set, we examine three different training/test regimes:
Other Domains (OD):
In this regime, when we test on problems from some domain, we train on all problems from all other domains.
This simulates the setting where we need to estimate search progress on a problem from a completely unknown domain.
Same Domain (SD):
In this regime, when we test on problems from some domain, we train only on problems from the same domain.
To have some meaningful learning, we only used domains with more than 15 instances, leaving us only 3 domains.
For each domain, we split its instances into the even- and odd-numbered problems, to obtain two sets of roughly the same size.
We then trained a predictor on one set, and evaluated its performance on the other, A ∗ /Lm-Cut A ∗ /HFF GBFS/Lm-Cut
GBFS/HFF
S   204 B 11/ 10/
12/20 M [Table] giving us 6 data points in total. Other Domains Tune Same (ODTS):
In this regime, we ﬁrst train a predictor based on instances from the other domains.
We then take this trained model, and ﬁne tune it on one half of the instances from the target domain (split, as before, to even and odd numbered problems).
During this ﬁne tuning we run the same training algorithm, except that the learning rate is 0.0001 and the batch size is 2048.
As ﬁne-tuning requires less data than SD, we used domains with more than 6 instances – giving us 6 domains and 12 data points in total.
We measure the accuracy of each search progress estimate prediction (for a given state) by the absolute error between the prediction and the true progress (which we have, since we use only solved problems).
The accuracy of a search progress estimator over a given problem is the average accuracy of its predictions over all the states expanded by the algorithm, that is the mean absolute error (MAE).
The accuracy for a set of problems (all problems in a domain, or all problems overall) is the average accuracy across these problems (giving equal weight to each problem, regardless of the number of states it required expending), that is, the average mean absolute error.
We also report the average of domain averages (that is, giving the same weight to each domain, regardless of how many problems it contains).
We remark here that we used MSE rather than MAE as the loss function during training, as MSE gives us a smoother function.
Nevertheless, as we discuss next, our technique achieves start-of-the-art performance [Table]
Empirical results
We now describe the results for our empirical evaluation.
First, Tables 2,3,4 and 5 compares our technique using the OD regime to the estimators proposed in previous work in all 4 conﬁgurations of heuristic and search algorithm.
There are some missing values in the table, as some planner conﬁgurations do not solve any problem for this domain.
We compare the previous work to OD, as it is the closest to these in terms of assumptions – it does not assume any knowledge about the problem it is used on.
To summarize these results more concisely [Table] Table 1 shows in how many domains OD was the best predictor, and the overall accuracy of our OD predictor vs. the best predictor from the previous work.
Overall, OD was the most accurate, with better average mean absolute error compared to the best predictor from previous work, VeSP/VaSP. The results when averaging over individual problems rather than domains are very similar. [Table] [Table]
[Table]
Domain
VaSP VeSP
PBP
DBP
OD Airport(6) 21.0(4.6) 38.8(8.9) 26.9(3.9) 37.3(7.8) 21.
[Table] (2.1) Blocks(20) 27.3(2.9) 30.9(5.3) 41.5(2.3) 35.5(7.0) 20.
[Table] [Table]
.5) Depot(19) 26. [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Elevators(27) 26.
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
.9) Freecell(14) 15.
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
.5) Gripper(4) 28. [Table] [Table] [Table]
[Table] [Table]
[Table]
Logistics(11) 24. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
.4) Miconic(5) 20.8(0.9) 37.9(
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
) Openstacks(6) 21. [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] .9) Parcprinter(8) 22.
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
To get a better sense of the behavior of these estimators during search, Figure 2(a,b,c) shows plots of the absolute error during search, for 3 problems of different sizes from different domains.
For each point on the x -axis, which indicates the true search progress, we plot the absolute error of each predictor at that state.
As we can see, VaSP has a lot of noise.
This is explained by the fact that every change in the value of h min causes a major change in the estimation.
With respect to the other estimators, our method had better accuracy until about the 60–70 percent point, and from there the prediction was least accurate and missed the end of the problem.
The difference between our model and the baselines at the end of the search is not surprising, as the baselines depend more closely on h , and thus when h is close to zero, the estimates predict we are close to end of search.
Having seen that our technique, in the OD regime, outperforms previous work, we now turn to evaluating the impact of more speciﬁc training data by comparing the OD regime to the SD and ODTS regimes.
Recall that in the SD regime our network is trained only on instances from the same domain, while in ODTS it is trained on instances from all other domains, and then ﬁne-tuned on the domain being evaluated.
Also recall that these regimes can only be used on domains which have enough data (15 solved instances for SD, 6 for
ODTS), and thus the comparison is on fewer domains.
Tables 6,7,8 and 9, shows the results for this evaluation.
[Table]
[Table] [Table] [Table]
using all 3 regimes [Table] In the table there are missing values as well, since in some conﬁguration the number of problems the planner solved was below the required number we deﬁne for this experiment.
Interestingly, ODTS outperforms SD in most cases, showing that the training data from other domains is important.
ODTS also outperforms OD in most domains as well as overall in each table, showing that domain-speciﬁc tuning does help.
Figure 3 plots the absolute error of the 3 different regimes on a speciﬁc instance, PSR-small 31.
As we can see, all 3 predictors started with unstable estimates, and got more and more accurate in the middle.
Near the end all predictors become less and less accurate and missed the fact they are near the goal.
Other than this, we can see that ODTS was generally more accurate, while OD and SD were each better than the other in different parts of the search. Generalization
We conclude our empirical evaluation by evaluating how our predictor handles changing the search algorithm, the heuristic, or both.
In this experiment, we trained our predictor (using the OD regime) on one choice of search algorithm and heuristic, and then tested its accuracy on another choice. With 2 search algorithms and 2 heuristics, we have 4 combinations, and thus 16 settings to evaluate. Table 10 shows the results of cross domain evaluation. As expected, using the same conﬁguration for training and in test leads to the best performance in 3 out of 4 cases.
The more interesting phenomenon is that when we change the algorithm and keep the heuristic the average MAE decreases by 0.86.
When we change the heuristic and keep the algorithm, MAE decreases by 1.18, and changing both decreases
MAE by 1.31.
Note that, for any conﬁguration, training on the polar opposite (different search algorithm and heuristic) still yields better accuracy than the best predictor from previous work.
This shows that our predictor can generalize to a different search algorithm and heuristic without suffering too severe a decrease in performance.
Discussion and Future Work
In this paper we have showed a novel approach to search progress estimation using deep learning.
We have shown that our approach outperforms previous state-of-the-art approaches, and that it beneﬁts from having access to better training data.
It is also interesting (and perhaps sad) to note that the previous search progress estimators (Thayer, Stern, and Lelis 2012), which were developed manually using extensive expertise in heuristic search and human creativity, are outperformed by a machine learning algorithm with very limited prior knowledge.
Besides being an interesting problem, search progress estimation can be used to make decisions about search.
For example, as previously mentioned, estimates of remaining search time have been used in situated temporal planning (Cashmore et al. 2018) and in Branch-and-Bound search (Anderson et al. 2019).
Other applications are in anytime search (Dionne, Thayer, and Ruml 2011) or metareasoning (Shperberg et al.
However, using search progress estimates to make decisions about search has a problem with self-reference – the predictions affect search decisions, which affect search time, which then affect the true search progress.
We intend to address this challenge in future work by attempting to learn a series of predictors which converge to a ﬁxed point which exhibits good performance.
Finally, we remark that we employed fairly simple machine learning tools here.
In future work we will examine whether we can improve performance even more by using different architectures of neural networks, or more informative features.