Paper: Chain-of-Symbol Prompting for Spatial Relationships in Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
The step (ii) in CoS prompting is not clear enough. How do authors correct the zero-shot CoT’s output, is it manually checking or prompting LLMs to do so?

Author response:
We manually correct zero-shot CoT's output, which described in Appendix B.2 .

Reviewer comment (Presentation):
There is an overlap between “Natural Language Navigation” and the text in the box in Figure 2.

Author response:
Thanks for your kindly mention! We will adjust it in our paper.

Reviewer comment (Experiments):
I am curious about the generalization ability of the proposed CoS prompting. I would encourage authors to report performance in more general domains rather than just spatial question answering, such as some open-domain QA tasks.

Author response:
In addition to spatial tasks, we add experiments on the temporal sequence QA tasks from the BBH benchmark which is shown in the third table of part 1 in general response for all reviewers.

Reviewer comment (Evaluation):
This paper presents very interesting findings, I would encourage authors to discuss more about its mechanisms and give some insights.

Author response:
Thank you for your encouragement. One intuition we think might be useful is that CoS uses symbols to replace part of natural languages, hence reducing noises brought from redundant natural languages. Neural network might only needs to know the relationship between entity A and other entities to conduct the complex reasoning or planning task. We postulate that removing irrelevant information contained in the natural language can help model to focus on the real task. And using symbols might can reduce potential context information gained from pre-training stage compared with natural languages.

Reviewer comment (Evaluation):
Though authors list some advantages of the proposed CoS, at the same time, CoS is more difficult to track LLMs reasoning path. I would encourage authors to discuss some limitations of this work.

Author response:
We will add more limitation about this aspect in our paper. Although our method uses less natural languages compared with CoT, we actually contains some information of input in CoS ( e.g. objects / entities ), which is easy to trace them back to input descriptions, which can then track the reasoning path.

Reviewer comment (Experiments):
The CoS prompting exemplars are generated with zero-shot CoT. I would encourage authors to discuss the motivation of selecting 0-shot CoT and its implications on CoS performance.

Author response:
Zero-shot CoT is the most straightforward clue that one could have to use CoS in their own application. While other format can be useful, the search space is huge and it can be spend more time in writing the demonstration fully manually. Also, we tried in our preliminary experiments on human-written CoS and CoT. We did not have promising performance on both CoS and CoT when human writting, and we postulate that LLMs adapts better to the zero-shot CoT format produced by themselves.

Reviewer comment (Novelty):
This is another prompting paper in the era of LLMs, and there is no new algorithm, but a trick (finding) to make the LLMs more robust.

Author response:
Although we don't propose new "algorithm", we provide insights to use symbols to replace natural languages in demonstration / prompting, which we think would be a promising direction to further explore for future works, for using symbols can wider the representation space in prompting, and reducing potential noises brought from natural languages. What's more, in our additional experiments about open-source LLMs, we find the scaling curve of CoS is steeper than CoT’s and the emergent ability of LLMs for understanding CoS (symbols) when size increases, which is very interesting. We also add different tasks of scaling curve in the latest uploaded version of the paper in figure 4, shows the general trend across tasks.

Reviewer comment (Theory):
What is the fundamental explanation that this method works well?

Author response:
We use CoS for two reasons:

Not only using natural languages but also using symbols can make the representation space wider in prompting.

CoS uses symbols to replace parts of natural languages, hence reducing noises brought from redundant natural languages. One assumption is that words of natural languages might have more context information during pre-training, which might bring potential additional information, but using symbols can avoid these, by connecting objects without potential additional information, and make prompting more clear and structural.

Reviewer comment (Experiments):
The process of converting spatial tasks into symbolic representations could introduce additional complexity and computational overhead. Besides, it require annotations, which seems more difficult to obtain compared with natural language based chain-of-thought or program-based program-of-thought.

Author response:
Based on guideline we give in the point 1 and in the section 3 of the paper, we can easily transform any sequential natural language exemplars which meet the conditions to chain-of-symbol exemplars. The guideline is suitable for all tasks which meet the conditions, we have proved its effectiveness in spatial relationships and temporal sequence reasoning tasks as examples, and future work can explore more tasks.

Reviewer comment (Experiments):
The open-source LLM remain under-explored, only ChatGPT series are tested in the experiments. It could have been more solid if the same trend can be observed in other LLMs. Besides, the design of prompt may also influence the results a lot, perhaps a through comparison on the robustness of the prompt should also be included, so that the CoT performance is really revealed in terms of capturing spatial relationship.

Author response:
For open-source models, the results and analyses are concluded in part 1 of general response for all reviewers. Figure 4 in our latest uploaded version of paper gives a clear results about scaling curves of CoS compared with CoT, it shows CoS has more steep curve for all tasks, which indicates it is a general ability that LLMs can understand abstract symbols better when size increases.

Reviewer comment (Experiments):
How does CoS perform when compared to other potential solutions or methods that might address spatial reasoning in LLMs? Such as program-based CoT that involve symbolic reasoning?

Author response:
For other methods, for example program-based CoT, can you be more specific? We only find this related work https://openreview.net/pdf?id=YfZ4ZPt8zd, which uses an external program executor to help calculate, but to be honest it is not a fair comparison with our CoS method which only does the inference by LLMs on their own, but we can list these kinds of works in our related work section.

Reviewer comment (Reproducibility):
How is the computation required to further convert to chain of symbols? And how about the tasks that do not have manually annotated symbols, how to extend your CoS method to a broader range of tasks?

Author response:
The computation required to convert symbols is down to 10ms through Python program based on given CoT exemplars. For those tasks which do not have manual annotated symbols, we can follow the procedure described in the part 1 of weakness above. For a broader range of tasks, just as the description in part 2 of general response for all reviewers, we explore temporal sequence tasks and verified the effectiveness of our method.

Reviewer comment (Experiments):
The paper focuses only on Text-Davinci-003 and GPT-3.5-turbo. However, it would be beneficial to include other promising open-source large language models like the Llama-2 series in their study to see the effect of model size, different base models, etc.

Author response:
You can see the part 1 of general response for all reviewers, we add the experiments of open-source LLMs, llama-2 with different sizes. And you can also see the figure 4 in latest uploaded version of our paper very clearly about the more steep scaling curve of our method across different tasks compared with CoT. It shows the effectiveness of our method compared with CoT and the enmergent ability of understanding CoS.

Reviewer comment (Evaluation):
The authors claim that they achieved a significant improvement of up to 60.8% in accuracy (from 31.8% to 92.6%) on the Brick World dataset. While this is technically correct given the results, the reported improvement is a little misleading because it's comparing zs-CoT vs. CoS and also such a dramatic gain isn't seen in most cases, especially when comparing the performance of CoS to CoT. In more realistic datasets like SPARTUN, the improvement is marginal. This suggests that the impressive gains are more about the simplicity of the Brick World dataset than any substantial leap in CoS performance. Thus, I believe the claim should be more moderate.

Author response:
Thanks for pointing out our misleading part, We revised our claim in the revision version. 

For the concern of marginal improvements, we add further experiments about using longer description instances (e.g. from 6 to 14 bricks) and find that CoS can gain more improvements compared with CoT in these scenarios compared with more simple scenarios, which is reported in part 2 of general response for all reviewers. 

For "the impressive gains are more about the simplicity of the Brick World dataset", actually the "shuffle both" results of brick 1d shows it is not a simple task as it looks like, where CoT only gains 43.0 accuracy. This actually shows that LLMs do not have good spatial understanding capabilities, because they cannot do spatial reasoning tasks which human think is easy well.

Reviewer comment (Evaluation):
Apart from the previously mentioned issues, I find the contribution is limited for the following reason: the paper does demonstrate how you can condense the chain of thought (CoT) into a series of symbols, and that this is enough for tasks involving spatial relationships. However, this method of prompting might only be effective for spatial relationship tasks. Also, when you look at the final experiment (Table 5), it suggests that this method might not work as well in real-life situations. It seems that the success of this method, referred to as CoS, might be limited to simpler, more straightforward spatial tasks.

Author response:
We add experiments on the temporal sequence tasks from the BBH benchmark [1]. This is another task which is not related with spatial relationships, and which also contains a more real-life situation. We report the results in table 2, part 1 of general response for all reviewers, and also tested on GPT-3.5-Turbo, and shows CoS can outperform CoT from 66.5 to 76.2.

"It seems that the success of this method, referred to as CoS, might be limited to simpler, more straightforward spatial tasks."
We use complexity control experiment and task of temporal sequence of BBH benchmark to shows the contribution of our proposed method, and we also want to describe our method in a more general way:

We will add this into Section 3 in our paper: we can formulate the tasks which our method is suitable as a set of tasks which has sequential and limited number of relationships, and the relationships can be spatial, and also can be other kinds such as temporal. 

For converting natural languages into symbols, we can use human, NER system or even rule-based codes to automatically recognise objects in the sequential relationships described by natural languages, and then connect them with each other using random symbols ( e.g. '/', '-', etc). By doing this, we actually use symbols to replace original relationships which connect objects together. 

Generally, we want to provide insights about using symbols in demonstration of the prompting, and we provide a particular way to use symbols to make LLMs to be more effective and efficient when handle sequential tasks.

Reviewer comment (Experiments):
The authors used GPT-4 exclusively for the final experiment. Why not present GPT-4 results in the other sections as well?

Author response:
We only have limited access of GPT-4 API currently, we will add the related results when we can access GPT-4 API in our future version.