Paper: Guiding Language Models Reasoning with Planning Tokens (ICLR-2024)

Reviewer comment (Experiments):
All the experiments are based on human annotated reasoning steps on math problems. It is unknown that how well the proposed method can generalize for automatically produced reasoning steps for general reasoning problems. Since the title is not specific to math problems, in order to achieve the wide-range claim in the title, it is necessary to investigate more general reasoning problems.

Author response:
2. **Title**: While our proposed method has great potential on applying to various kinds of reasoning, our empirical results are currently focused on the math word problem datasets. We will restrict our title to math reasoning by changing it to: Guiding Language Models Math Reasoning with Planning Tokens

Reviewer comment (Experiments):
When doing cluster for each reasoning step, the contextual information of each step seems not explored. In section 2.3.2, to obtain the representation vector, each reasoning step is separately encoded. The authors should consider a contextual version, which use the prefix reasoning steps and the question context to generate the representation vectors of the current step.

Author response:
3. **Contextual information of each step**: We actually have already taken the contextual information into consideration, by using the contextualized embedding of each reasoning step produced by the base LLM. More specifically, we concatenate the question and reasoning steps, and encode the whole sequence with the base LLM. We then average the contextual embeddings of all tokens in a reasoning step. This makes the embedding of each reasoning step conditioned on the previous ones. We will clarify this important detail in the paper.

Reviewer comment (Novelty):
Soft-VAE is not novel. Why not simply using VQ-VAE?

Author response:
4. **VQ-VAE v.s. SQ-VAE**: We actually tried VQ-VAE first, and found it is very unstable to train on the step embeddings, so we switched to a soft version of it. We agree that this soft version of VQ-VAE is not a significantly novel contribution. Instead, this is an implementation choice for our proposed method.

Reviewer comment (Experiments):
Do you do the clustering based on the training set or just do the clustering using the test set?

Author response:
1. **Clustering**: We only do clustering on the training set. The testing set is not used in our learning process.

Reviewer comment (Experiments):
The optimal setting of the number of cluster can be varied across datasets. It is better to also perform ablation studies on the other dataset.

Author response:
3. **Ablation on all datasets**: We would love to provide them on all datasets, but we weren’t able to do so because of our time and compute constraints, since these experiments are expensive to run.

Reviewer comment (Evaluation):
The performance on the MATH dataset is notably lower than the others.

Author response:
Thank you for your positive review!

The MATH dataset is significantly more difficult compared to the other two datasets. It remains challenging for very strong LLMs, such as GPT-4.

* Data source: The MATH dataset comes from the AMC math competitions. In contrast, GSM8K comes from middle school and high school math exams; AQUA comes from multiple choice GRE math tests. 
* Answer format: The answers of MATH data are free-form text, e.g., Latex expressions such as \frac{1-\sqrt{2}}{2}, which makes it hard for the model to get every bit of the answer correctly. In contrast, the answers of GSM8K are single int numbers; the answers of AQUA are single characters, indicating one of the choices.