Paper: Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-Temporal Reasoning (ICLR-2024)

Reviewer comment (Novelty):
Some references on LLM + planning are missing:  
[1] DEPS: https://arxiv.org/abs/2302.01560  
[2] Plan4MC: https://arxiv.org/abs/2303.16563  
[3] GITM: https://arxiv.org/abs/2305.17144

Author response:
**Missing references**  
We thank the reviewer for suggesting the missing references! We apologize for the oversight and missing these references; we will make sure to add them to our updated draft. 

Looking into the suggested papers, they all revolve around planning in a minecraft environment. Below, we briefly discuss their connect to our work.  
[1] proposed an interactive approach for open-world planning based on LLMs allowing error correction through self-explanation. This, indeed is relevant to our work as we find substantial improvements on our benchmark by incorporating “interactive” environment feedback (i.e., the ReAct experiments).

[2] relied on an approach where LLMs are used for skill planning. The proposed approach combines reinforcement learning with LLMs in order to decompose the planning task in a minecraft world to achieve long-term planning tasks. LLMs are used to build a skill graph by learning relationships between skills, while RL is used to plan over the learnt graph. This paper highlights another example of how LLMs can be useful tools to assist with long-term planning approaches. 

[3]  Used LLM to break down the goals and map them to structured actions for final control signals. The proposed LLM-based agent consists of an LLM Decomposer, an LLM Planner, and an LLM Interface. Similarly to [1], this work is very relevant to our work as it supports the conclusions reached through our ReAct agent.

Reviewer comment (Experiments):
In Table 2, there are some results on OOD evaluation with in-context learning. Can the authors clarify the exact settings of this? What are the data used to fine-tune the models, and what are the in-context examples when performing the OOD evaluations?

Author response:
**Q1: Clarification on the OOD evaluation**  
For the training data for the fine-tuned models, we use the “#Train” subsets from Table 1, which contains 16k instances in single-goal setting and 53k instances in multi-goal setting, all under 6x6 grids and with 1-5 obstacles. We point the reviewer to the first as well as the last two paragraphs of Section 3 for implementation details and experimental design. for more details about the different settings. Similarly, for in-context learning, we draw few-shot examples from the training sets used for finetuning and evaluate on instances drawn from the OOD test sets.

Reviewer comment (Reproducibility):
What are the exact prompts used in the task? Specifically, what is the prompt for predicting whether the goal is reachable?

Author response:
**Q2: Clarification on the “exact prompts used in the task”**  
The full prompts used for all in-context experiments are presented in Appendix B for reproducibility. The few shot exemplars are drawn from the training set. The prompt for predicting whether the goal is reachable is not a separate one, however, we provide few-shot exemplars that include this case, and we expect the LLMs to identify these cases based on these examples.

Reviewer comment (Experiments):
My biggest concern is that the experiment setting is a bit too simple: it's just a set of discrete 2D grid, which is far from ideal and realistic path planning setting: high DoF, 3D space, continuous action. I understand 2D grid is a good starting point, but still, it doesn't provide sufficient value for revealing deep enough insight into LLM's limits. For example, such experiments don't shed light on how current LLMs can reason in 3D space.

Author response:
We thank the reviewer for sharing the thoughtful comments on our work! We appreciate the reviewer’s acknowledgement on the importance of our research topic (i.e., temporal and spatial reasoning). 

We refer the reviewer to “General Responses to All Reviewers” regarding the concern about “the experiment setting is a bit too simple” and “multimodal” modeling. We welcome further comments from the reviewer and will be happy to have deeper discussions!

Reviewer comment (Experiments):
This is a bit philosophical: spatial reasoning in a blind (pure language) space is, at least to me, not a well grounded request. I understand at the time of submitting, GPT-4v is not available yet, but there are also other large multimodal available, such as Bard. Maybe such experiments would be more justified if the reasoning is grounded with a vision input? After all, spatial path planning with only access to language description, even for humans, is not a very common task. I would like to see more insights on this from the authors.

Author response:
We thank the reviewer for sharing the thoughtful comments on our work! We appreciate the reviewer’s acknowledgement on the importance of our research topic (i.e., temporal and spatial reasoning). 

We refer the reviewer to “General Responses to All Reviewers” regarding the concern about “the experiment setting is a bit too simple” and “multimodal” modeling. We welcome further comments from the reviewer and will be happy to have deeper discussions!

Reviewer comment (Experiments):
Another one of the concerns about the paper is that the domain studied, such as 7x7 path planning, could be considered somewhat simplistic or "toy-like." This raises questions about the extent to which the findings can be generalized to more complex, real-world scenarios.

Author response:
**Q1: How do the authors justify the use of the 7x7 path planning environment as a valid proxy & Q2: findings are scalable and applicable to real-world tasks**

We refer the reviewer to the general response for the discussion about PNNL’s simplicity but that its findings can be generalized to more complex, real-world scenarios.

Reviewer comment (Experiments):
The use of small-scale environments may not adequately capture the challenges and nuances that would be present in larger, more intricate settings that LLMs might encounter in practical applications.

Author response:
**Q1: How do the authors justify the use of the 7x7 path planning environment as a valid proxy & Q2: findings are scalable and applicable to real-world tasks**

We refer the reviewer to the general response for the discussion about PNNL’s simplicity but that its findings can be generalized to more complex, real-world scenarios.

Reviewer comment (Experiments):
If the benchmarking tasks do not accurately reflect the complexity of real-world tasks, it may limit the utility of the findings.

Author response:
**Q1: How do the authors justify the use of the 7x7 path planning environment as a valid proxy & Q2: findings are scalable and applicable to real-world tasks**

We refer the reviewer to the general response for the discussion about PNNL’s simplicity but that its findings can be generalized to more complex, real-world scenarios.

Reviewer comment (Evaluation):
It seems to me that the results here could be explained in an entirely different way.

Author response:
We are grateful to the reviewer for their insightful comment on the state of research on LLM spatial reasoning! 

We noticed a misunderstanding on our experimental results. Therefore, we would first like to clarify our major observation. That is, based on our results, we believe that LLMs do exhibit __some__ ability for reasoning over spatial information. However, they are still __not perfectly__ solving spatial reasoning, and they have failed in long-horizon temporal reasoning. 

Below, we clarify how we reached the conclusion that “LLMs do exhibit __some__ ability for spatial reasoning”. We believe that the observed success of LLMs is not solely “pattern/instruction imitation”. Whether LLMs can reason or not is still a debatable question and our work does not aim to resolve the argument (instead, we provided a benchmark to facilitate it). In our humble opinion, pattern imitation happens when the same or similar patterns have been provided to the LLM, either during its pre-training or in-context learning, so that the LLM may succeed by memorizing the patterns from the data and then adapting the language to the specific test input, rather than reasoning about the test problem from scratch. However, it is important to note that the path planning task we have focused on in this work is very different from the commonly seen math reasoning tasks (e.g., GSM8k, MATH) in the following ways:

First of all, path planning itself is an optimization problem. As elaborated in Appendix A.2, a path planning task essentially projects to the A* search (single-goal setting) or combining A* search with the Traveling Salesman (multi-goal setting) problems. This is very different from math problem solving where an LLM could cheat by imitating the pattern of how numbers given in the condition can be combined into an equation and eventually led to a numerical answer. Due to its optimization nature, path planning cannot be achieved with trivial pattern imitation.

While math problems are often framed in daily scenarios (such that an LLM may leverage the commonsense pattern it has learned to play “shortcuts”), the path planning problems in PNNL is highly symbolic (e.g., using p0 and p1 rather than the specific city names as locations, and indicating all obstacles using x-y coordinates). So playing “shortcuts” is not very likely.
We want to clarify that our prompts to LLMs do not include the exact test environments, and can even include completely OOD ones, so there does not exist any potential “pattern” for imitation. Notably, GPT-4 with CoT obtains stable generalization to smaller or larger grid environments (success rate of 0.787 -> 0.763 and 0.836). While it suffers from degradation in the obstacle OOD evaluation, the success rate is still decent (0.544). To further uncover its underlying mechanism, we added one experiment for 168 20x20 grid environments, each consisting of 40 obstacles. We still observed a success and optimal rates of 0.279 from GPT-4 AE and 0.351 from GPT-4 CoT. This implied that LLMs do reason about their optimization strategy in path planning, although the strategy is not very generalizable. We would like to denote a distinction between (a) whether LLMs can reason and (b) where their reasoning outcome (e.g., their path planning strategy) is generalizable to the more complicated scenarios. Intuitively, even for humans (e.g., human soldiers), when they have learned skills exclusively only in a small-size practice room, they would have difficulty when entering a real-size battlefield, but this does not imply their lack of reasoning or learning during the practice.
Lastly, while similar math problems may have potentially been included the pre-training data of LLMs, this is much less likely to be true for path planning, because intuitively people do not commonly verbalize a symbolic task on the Internet data. 

As discussed in our experimental results section, LLMs still cannot “perfectly” reason about the spatial information. This again can be evidenced by the degradation of LLMs in OOD evaluation. They also fail to perform long-horizon, temporal reasoning; for example, we observed that the success of ReAct compared with CoT is attributed to the way how it receives environment feedback and decomposes a long-horizon task into multiple short-horizon ones. Another result highlighting LLMs inability for temporal planning is the low ability to find the optimal path, particularly for the multi-goal setting; We particularly probe the LLM ability to perform optimization in Appendix D, and present the results in Table 10, which clearly shows that GPT-4 falls short on this task.

Reviewer comment (Evaluation):
Even though there are systematically constructed evaluations of increasingly complex path planning problems across different dimensions (length, number of obstacles, etc.), I do not see how the fact that LLMs fail on the more complex tasks does not just imply that the pattern-based instruction imitation of LLMs (the alternative interpretation of LLM instruction success) does not just fall apart more quickly on more difficult tasks. Imitating textual examples of instructions that these LLMs have been trained on in a pattern-based way would yield significant success in providing directional information, particularly over short horizons. Difficult tasks (long horizon, etc.) which require spatial reasoning have a lower probability of accidentally being successful with imitation-based responses.

Author response:
We are grateful to the reviewer for their insightful comment on the state of research on LLM spatial reasoning! 

We noticed a misunderstanding on our experimental results. Therefore, we would first like to clarify our major observation. That is, based on our results, we believe that LLMs do exhibit __some__ ability for reasoning over spatial information. However, they are still __not perfectly__ solving spatial reasoning, and they have failed in long-horizon temporal reasoning. 

Below, we clarify how we reached the conclusion that “LLMs do exhibit __some__ ability for spatial reasoning”. We believe that the observed success of LLMs is not solely “pattern/instruction imitation”. Whether LLMs can reason or not is still a debatable question and our work does not aim to resolve the argument (instead, we provided a benchmark to facilitate it). In our humble opinion, pattern imitation happens when the same or similar patterns have been provided to the LLM, either during its pre-training or in-context learning, so that the LLM may succeed by memorizing the patterns from the data and then adapting the language to the specific test input, rather than reasoning about the test problem from scratch. However, it is important to note that the path planning task we have focused on in this work is very different from the commonly seen math reasoning tasks (e.g., GSM8k, MATH) in the following ways:

First of all, path planning itself is an optimization problem. As elaborated in Appendix A.2, a path planning task essentially projects to the A* search (single-goal setting) or combining A* search with the Traveling Salesman (multi-goal setting) problems. This is very different from math problem solving where an LLM could cheat by imitating the pattern of how numbers given in the condition can be combined into an equation and eventually led to a numerical answer. Due to its optimization nature, path planning cannot be achieved with trivial pattern imitation.

While math problems are often framed in daily scenarios (such that an LLM may leverage the commonsense pattern it has learned to play “shortcuts”), the path planning problems in PNNL is highly symbolic (e.g., using p0 and p1 rather than the specific city names as locations, and indicating all obstacles using x-y coordinates). So playing “shortcuts” is not very likely.
We want to clarify that our prompts to LLMs do not include the exact test environments, and can even include completely OOD ones, so there does not exist any potential “pattern” for imitation. Notably, GPT-4 with CoT obtains stable generalization to smaller or larger grid environments (success rate of 0.787 -> 0.763 and 0.836). While it suffers from degradation in the obstacle OOD evaluation, the success rate is still decent (0.544). To further uncover its underlying mechanism, we added one experiment for 168 20x20 grid environments, each consisting of 40 obstacles. We still observed a success and optimal rates of 0.279 from GPT-4 AE and 0.351 from GPT-4 CoT. This implied that LLMs do reason about their optimization strategy in path planning, although the strategy is not very generalizable. We would like to denote a distinction between (a) whether LLMs can reason and (b) where their reasoning outcome (e.g., their path planning strategy) is generalizable to the more complicated scenarios. Intuitively, even for humans (e.g., human soldiers), when they have learned skills exclusively only in a small-size practice room, they would have difficulty when entering a real-size battlefield, but this does not imply their lack of reasoning or learning during the practice.
Lastly, while similar math problems may have potentially been included the pre-training data of LLMs, this is much less likely to be true for path planning, because intuitively people do not commonly verbalize a symbolic task on the Internet data. 

As discussed in our experimental results section, LLMs still cannot “perfectly” reason about the spatial information. This again can be evidenced by the degradation of LLMs in OOD evaluation. They also fail to perform long-horizon, temporal reasoning; for example, we observed that the success of ReAct compared with CoT is attributed to the way how it receives environment feedback and decomposes a long-horizon task into multiple short-horizon ones. Another result highlighting LLMs inability for temporal planning is the low ability to find the optimal path, particularly for the multi-goal setting; We particularly probe the LLM ability to perform optimization in Appendix D, and present the results in Table 10, which clearly shows that GPT-4 falls short on this task.

Reviewer comment (Evaluation):
The fine-tuning results, where improvement is found in distribution and failure occurs on out-of-distribution examples, seem to support this alternate interpretation — not the authors’ interpretation.

Author response:
We thank the reviewer for checking our response, but there still seems to be a misunderstanding on our results. **We want to clarify that none of the experimental result tables should be read separately. That is, the performance of an LLM should be understood based on its performance on all the evaluation settings (IID + OOD).** For example, we never claimed any of the fine-tuned or few-shot LLMs have “solved” spatial-temporal reasoning because of the observations that the reviewer has also noticed, i.e., they still suffer from performance degradation in OOD generalization, despite the good IID performance. To make it clear, in our next version, we will introduce a single metric that consolidates the observations from both IID and OOD, so this more fair and comprehensive evaluation would not be mistaken.


> “playing shortcuts is not very likely”

Exact match scores can, in fact, support this claim – If the LLM was merely following the exact patterns presented in the few shot example (i.e. inferring the exact process by which those exemplars were generated), we would expect the values for the exact match metric to be almost identical to the success rate (i.e. every time the model succeeds at finding a path, it is exactly identical to the gold truth). However, we notice a significant gap between these two metrics, particularly in the multi-goal setting (see Table 4)

> “If LLMs are not able to perform any mature level of spatial-temporal reasoning, why should we use a benchmark to measure spatial-temporal reasoning where 90+% success metrics reported in results tables for models that cannot do this reasoning? See Table 3 in the paper.”


Again, we note that none of our result tables should be read separately as they can give a comprehensive and correct interpretation of an LLM’s performance only when they are read together. Therefore, seeing a 90% success rate on a single table does not confirm any signals of true capability.
In addition, the reviewer seemed to misread our results. In table 3, 90%+ metrics are reported for: 1) fine-tuned models on the 5x5 grids; we explain this by the fact that 5x5 grids are inherently part of the 6x6 grids, hence these models were able to navigate them, without having to go out of distribution since all of the coordinates exist in the 6x6 grids, and 2) ReAct, which is not sensitive to the grid sizes as it solves the problem “incrementally” by decomposing it into multiple subtasks.



We hope our follow-up response can resolve the reviewer’s confusion on how to read our result tables. Furthermore, as the reviewer has noticed, how well LLMs can reason requires comprehensive experiments and evaluations from different angles (e.g., IID+OOD). Our main contribution, i.e., the PNNL benchmark, is developed to contribute to such discussions. We believe our benchmark, including its potential extension in the future (e.g., environment size, geometric shapes, evaluation metrics, etc.), will inspire further discussions around this topic.

Reviewer comment (Novelty):
Even in the spatial reasoning section of the related work, it does not appear that any prior work supports the idea that spatial reasoning can be performed by LLMs. There are three types of work cited by the authors which also represent my understanding of the community's view of this problem:  
1. LLMs can be used via methodological automated prompting to develop spatial plans (the prompting method + LLM executes planning which has varying success in accomplishing the task),  
2. LLMs have some level of spatial understanding (textual request for code to make images has significant success, etc.),  
3. LLMs do not perform reasoning for math/planning/etc. problems — just quite good mimicry.

Author response:
In summary, our intention of the experiments is to systematically study the state of reasoning capability of LLMs when they are augmented with advanced prompting methods, but our results do not imply mature spatial-temporal reasoning of LLMs. We apologize for any confusing language in our result discussion, but we would also like to emphasize that whether LLMs can reason or not, as the reviewer pointed out, is still an open question. Our main goal of this research is to provide a better benchmark to facilitate this exploration but not to resolve it.

Reviewer comment (Evaluation):
If the authors can explain and convince me that these results show LLMs actually perform reasoning over spatial information during the rebuttal, I would be willing to significantly increase my score.

Author response:
We are grateful to the reviewer for their insightful comment on the state of research on LLM spatial reasoning! 

We noticed a misunderstanding on our experimental results. Therefore, we would first like to clarify our major observation. That is, based on our results, we believe that LLMs do exhibit __some__ ability for reasoning over spatial information. However, they are still __not perfectly__ solving spatial reasoning, and they have failed in long-horizon temporal reasoning. 

Below, we clarify how we reached the conclusion that “LLMs do exhibit __some__ ability for spatial reasoning”. We believe that the observed success of LLMs is not solely “pattern/instruction imitation”. Whether LLMs can reason or not is still a debatable question and our work does not aim to resolve the argument (instead, we provided a benchmark to facilitate it). In our humble opinion, pattern imitation happens when the same or similar patterns have been provided to the LLM, either during its pre-training or in-context learning, so that the LLM may succeed by memorizing the patterns from the data and then adapting the language to the specific test input, rather than reasoning about the test problem from scratch. However, it is important to note that the path planning task we have focused on in this work is very different from the commonly seen math reasoning tasks (e.g., GSM8k, MATH) in the following ways:

First of all, path planning itself is an optimization problem. As elaborated in Appendix A.2, a path planning task essentially projects to the A* search (single-goal setting) or combining A* search with the Traveling Salesman (multi-goal setting) problems. This is very different from math problem solving where an LLM could cheat by imitating the pattern of how numbers given in the condition can be combined into an equation and eventually led to a numerical answer. Due to its optimization nature, path planning cannot be achieved with trivial pattern imitation.

While math problems are often framed in daily scenarios (such that an LLM may leverage the commonsense pattern it has learned to play “shortcuts”), the path planning problems in PNNL is highly symbolic (e.g., using p0 and p1 rather than the specific city names as locations, and indicating all obstacles using x-y coordinates). So playing “shortcuts” is not very likely.
We want to clarify that our prompts to LLMs do not include the exact test environments, and can even include completely OOD ones, so there does not exist any potential “pattern” for imitation. Notably, GPT-4 with CoT obtains stable generalization to smaller or larger grid environments (success rate of 0.787 -> 0.763 and 0.836). While it suffers from degradation in the obstacle OOD evaluation, the success rate is still decent (0.544). To further uncover its underlying mechanism, we added one experiment for 168 20x20 grid environments, each consisting of 40 obstacles. We still observed a success and optimal rates of 0.279 from GPT-4 AE and 0.351 from GPT-4 CoT. This implied that LLMs do reason about their optimization strategy in path planning, although the strategy is not very generalizable. We would like to denote a distinction between (a) whether LLMs can reason and (b) where their reasoning outcome (e.g., their path planning strategy) is generalizable to the more complicated scenarios. Intuitively, even for humans (e.g., human soldiers), when they have learned skills exclusively only in a small-size practice room, they would have difficulty when entering a real-size battlefield, but this does not imply their lack of reasoning or learning during the practice.
Lastly, while similar math problems may have potentially been included the pre-training data of LLMs, this is much less likely to be true for path planning, because intuitively people do not commonly verbalize a symbolic task on the Internet data. 

As discussed in our experimental results section, LLMs still cannot “perfectly” reason about the spatial information. This again can be evidenced by the degradation of LLMs in OOD evaluation. They also fail to perform long-horizon, temporal reasoning; for example, we observed that the success of ReAct compared with CoT is attributed to the way how it receives environment feedback and decomposes a long-horizon task into multiple short-horizon ones. Another result highlighting LLMs inability for temporal planning is the low ability to find the optimal path, particularly for the multi-goal setting; We particularly probe the LLM ability to perform optimization in Appendix D, and present the results in Table 10, which clearly shows that GPT-4 falls short on this task.