Paper: On Using Admissible Bounds for Learning Forward Search Heuristics (ICLR-2024)

Reviewer comment (Experiments):
The choice of using Greedy Best-first Search (GBFS) needs more justification as it is not guaranteed to find optimal solutions. Can the authors explain why they did not choose to experiment with A*?

Author response:
The reason for choosing GBFS over A\* is that the learned heuristic $\hat{h}$
is not guaranteed to be **admissible**, regardless of the ML method employed
(NLM, LR or HGN) and how the heuristic is modelled ($\mathcal{N}$ vs $\mathcal{TN}$).
An admissible heuristic is one that never exceeds the optimal plan cost
$h^*$ for any state: $\forall s \in S, \hat{h}(s) \leq h^*(s)$.
Unlike symbolic algorithms, machine learning methods (e.g., neural networks) do not provide such guarantees in general.
It is important to note that this issue is not particular to our approach since, to the best of our knowledge, **all heuristics learned by ML methods lack admissibility**.

The A\* algorithm is guaranteed to return an optimal plan *only when* the heuristic used to guide the search is admissible (Russell and Norvig, 2010, pp. 94-95). Since $\hat{h}$ is inadmissible, we cannot guarantee solution optimality regardless of the search algorithm employed (A\* or GBFS).
Therefore, we disregard plan length altogether and evaluate search performance as the number of node evaluations needed to find a solution, i.e.,
focus on satisficing planning.

In satisficing planning, A\* is known to often be slower than WA\* and GBFS (corresponding to WA\* with $W\rightarrow\infty$) because it needs to explore all the nodes below the best current $f=g+h$ value, while WA\* and GBFS do not.
This is the main reason why our work and many others in the heuristic learning literature (Ferber et al., 2022; Gehring et al., 2022; Yoon et al., 2008) employ GBFS instead of A*. **A* is not an appropriate baseline for the satisficing setting.**

Reviewer comment (Experiments):
A popular direction for evaluating the effectiveness of a learned heuristic is to test how well it generalizes to larger instances of a class of problems. The current evaluations focus only on problems generated with the same set of parameters. Adding this kind of generalization experiment would demonstrate the practical value of the proposed approach better.

Author response:
We have performed new experiments where we test the generalization abilities of the learned heuristics on larger problems. An overall analysis of the results obtained can be found in the common answer to all reviewers.

As requested, we updated the paper with new results and analysis (table S15, see page 28, Appendix H) that shows the planning results also generalizes to larger instances.

We did not include it in the first reply due to the time constraints: we were only able to run experiments for the NLM model and without FF-based tiebreaking (new enhancements added in the first reply). We are also running the other configurations (HGN, LR, with/without tiebrekaing, on large instances).

The new results support our claim that $\mathcal{TN}$+NLM is able to generalize to larger problems.
The (relative) results on large problems are almost identical to those obtained for small problems: $\mathcal{TN}$ outperforms $\mathcal{N}$ and $\mathcal{N}+clip$ in 4 out of 6 cases, with one case obtaining similar performance to $\mathcal{N}+clip$ and being outperformed by $\mathcal{N}$ in blocksworld with fixed/none (this also occurred for small problems, since this is a problematic configuration due to the fixed $\sigma$ and no residual learning). When comparing to the FF baseline, the results are also the same: $\mathcal{TN}$ greatly outperforms FF in blocksworld, obtains similar results in logistics and is outperformed in satellite.

Given that the results for small and large problems are very similar, we believe that, just as it occurred for small problems, employing FF-based tiebreaking will allow out approach to outperform the FF baseline on large problems for all three domains. We will add the missing experiments (tiebreaking and results for HGN and LR) as soon as we have time to run these experiments.

Reviewer comment (Experiments):
The three classical planning problems should have sufficiently detailed descriptions.

Author response:
We have added detailed descriptions for the three domains (blocksworld, logistics and satellite) used in our paper, along with their PDDL encodings.
This information can now be found in Appendix I.

Reviewer comment (Experiments):
Please see the above section for my question regarding the choice of the search algorithm.

Author response:
The reason for choosing GBFS over A\* is that the learned heuristic $\hat{h}$
is not guaranteed to be **admissible**, regardless of the ML method employed
(NLM, LR or HGN) and how the heuristic is modelled ($\mathcal{N}$ vs $\mathcal{TN}$).
An admissible heuristic is one that never exceeds the optimal plan cost
$h^*$ for any state: $\forall s \in S, \hat{h}(s) \leq h^*(s)$.
Unlike symbolic algorithms, machine learning methods (e.g., neural networks) do not provide such guarantees in general.
It is important to note that this issue is not particular to our approach since, to the best of our knowledge, **all heuristics learned by ML methods lack admissibility**.

The A\* algorithm is guaranteed to return an optimal plan *only when* the heuristic used to guide the search is admissible (Russell and Norvig, 2010, pp. 94-95). Since $\hat{h}$ is inadmissible, we cannot guarantee solution optimality regardless of the search algorithm employed (A\* or GBFS).
Therefore, we disregard plan length altogether and evaluate search performance as the number of node evaluations needed to find a solution, i.e.,
focus on satisficing planning.

In satisficing planning, A\* is known to often be slower than WA\* and GBFS (corresponding to WA\* with $W\rightarrow\infty$) because it needs to explore all the nodes below the best current $f=g+h$ value, while WA\* and GBFS do not.
This is the main reason why our work and many others in the heuristic learning literature (Ferber et al., 2022; Gehring et al., 2022; Yoon et al., 2008) employ GBFS instead of A*. **A* is not an appropriate baseline for the satisficing setting.**

Reviewer comment (Evaluation):
Additionally, can the authors add the standard deviation numbers for the average number of node evaluations to Table 2?

Author response:
We have added the std of node evaluations, as requested.
This new information can be found in Appendix F.

Reviewer comment (Experiments):
Missing state-of-the-art recent baseline: [Chrestien et al., 2022] is an alternative approach that also argues against using MSE and proposes an alternative approach. This baseline should be compared to the proposed approach for learning heuristics.

Author response:
Thank you for highlighting this interesting work. We would like to note that (Chrestien et al., 2022) follows the learning-to-rank approach initiated by (Garrett et al., 2016) which learns the relative quality between states, with an additional loss for enforcing heuristic monotonicity. We believe the method proposed in (Chrestien et al., 2022) corresponds to a learning-to-rank approach because the proposed L\* loss function tries to rank the states expanded by A\* before those which are not (as well as ranking them according to monotonicity), modeling the learning task as a binary classification rather than a regression as in our work.

We want to emphasize that **learning-to-rank methods deviate significantly from the cost-to-go learning literature** (which our work belongs to), both empirically and theoretically, as they pursue a different goal (correct node ordering vs correct cost-to-go estimation) and, thus, are significantly outside the scope of our work.
Our paper aims at addresssing the lack of formal understanding in the cost-to-go learning literature and, in section 3, paragraph 8, we explicitly state that our goal is to learn the cost-to-go $p^*(x|s)$.

Learning-to-rank methods directly optimize search performance by learning the node ordering. On the other hand, cost-to-go approaches learn to estimate the distance from some particular state to the goal, a unique capability missing from ranking-based methods. Being able to estimate the cost-to-go is useful in many practical applications, e.g., knowing the estimated travel time to destination in a mobile navigation app (e.g., Google Maps), as opposed to just
knowing which direction to go at every intersection (equivalent to ranking the directions).

Applying our theoretical intuition from $\mathcal{TN}$ to the learning-to-rank approach
is an interesting direction for future work.

Reviewer comment (Evaluation):
There is no analysis of performance vs. problem size. It would be very useful to see if the patterns depend on problem size.

Author response:
> There is no analysis of performance vs. problem size. It would be very useful to see if the patterns depend on problem size.

We have performed new experiments where we test the generalization abilities of the learned heuristics on larger problems. An overall analysis of the results obtained can be found in the common answer to all reviewers.

Reviewer comment (Theory):
Minor point: in Section 3, the description of learning heuristic ignores the goal, the learning of heuristics is conditioned on the goal state since the same state will have different estimate conditioned on different states.

Author response:
> in Section 3, the description of learning heuristic ignores the goal

Thank you for bringing this oversight to our attention. It has now been corrected in the revision, by stating that heuristics depend on both states and goals.

Reviewer comment (Novelty):
In my view, one of the main contributions of this paper is to use the NLL as their training loss instead of MSE, and NLL adds the prediction of $\sigma$ (the variance) where MSE uses the fixed $\sigma$. However, in my opinion, this technique will improve the performance very trivially since the model will predict better with more parameters.

Author response:
### Re: Weaknesses: More parameters?

> NLL adds the prediction of $\sigma$ ... the model will predict better with more parameters.

Our improvement is not due to an increase in the number of parameters.
In order to predict $\sigma$, the neural network (NLM) needs to have two outputs, but this only affects the size of the last linear layer of the network, which is negligible compared to the size and parameters of the entire network.
Moreover, in our preliminary experiments, doubling the total number of model parameters (NLM width) did not significantly affect the performance.

The reason why our NLL loss outperforms the standard MSE is due to the underlying modelling assumptions:
standard MSE models $h^*$ as $\mathcal{N}(\mu,1/\sqrt{2})$, which is bad;
the "MSE with learned sigma" models it as $\mathcal{N}(\mu,\sigma)$, which is better but still bad;
finally, our NLL models it as $\mathcal{TN}(\mu,\sigma,lb=h^{\downarrow},\infty)$, which is the best model both empirically and theoretically (max-ent distribution).
This is easy to check by looking at Table 1 in our main paper. It can be observed that $\mathcal{TN}$ for the _fixed/none_ and _fixed/_ $h^{FF}$ configurations obtains better accuracy (_MSE_ rows) than $\mathcal{N}$ for _learn/none_ and _learn/_ $h^{FF}$.
This means that an ML model that only predicts $\mu$ (_fixed_ configurations) and is trained with our custom NLL outperforms another model that predicts both $\mu$ and $\sigma$ (_learn_ configurations) but is trained with the standard MSE.

We believe our answers address all your questions and concerns.

Reviewer comment (Evaluation):
In section 3, the authors claim that the importance of using NLL loss function instead of MSE. However, from the experiment results, the authors can only claim that $\mathcal{T} \mathcal{N}$ obtains around 40 percent lower MSE than $\mathcal{N}$ +clip on (geometric) average, so does this truncated Gaussian technique play the major role of the outstanding performance of the experiment instead of the choice of loss function? And what is the statistical intuition behind it?

Author response:
### Re: Question 1

Firstly, we would like to clarify a point of a slight confusion.
Your comment mentions as follows:

> In my view, ... to use the NLL as their training loss **instead of** MSE, ...

> In section 3 ... the importance of using NLL loss function **instead of** MSE.

Let us clarify that "use NLL **instead of** MSE" involves a slight misnomer,
as the MSE is indeed a NLL for $\mathcal{N}(\mu, \sigma=1/\sqrt{2})$.
In our paper, we wrote:

> [Main paper, Section 2, Paragraph 5] We emphasize that the choice of the distribution determines the loss.

This means that there is a 1-to-1 correspondence between the distribution used and the loss to minimize, i.e.,
they are **_two sides of the same coin_**.
Therefore, as to the question

> ... does this truncated Gaussian technique play the major role ... instead of the choice of loss function?

We answer that _the choice of our loss function is a direct consequence of choosing the truncated Gaussian distribution_,
therefore it is not possible to choose them separately.
For example, using a truncated Gaussian while minimizing a standard MSE loss
is impossible/contradicting
(minimizing MSE implies using $\mathcal{N}(\mu, \sigma=1/\sqrt{2})$).

Likewise, switching modelling assumptions during training and evaluation
(e.g., train the model with $\mathcal{N}(\mu, \sigma)$, then reuse the network for $\mathcal{TN}(\mu, \sigma, l, \infty)$) is highly ad-hoc and should not be done.


In addition, regarding this comment:

> the authors can only claim that $\mathcal{TN}$ obtains around 40 percent lower MSE than $\mathcal{N}+clip$ on (geometric) average, ...

we would like to mention that predictions from the $\mathcal{N}+clip$ model are never worse than those of $\mathcal{N}$, since $\mathcal{N}+clip$ prevents incorrect
heuristic predictions that are smaller than the admissible heuristic.
Therefore, we can not only claim that $\mathcal{TN}$ is better than $\mathcal{N}+clip$, but also claim that it is better than $\mathcal{N}$ too.