Paper: Let's Verify Step by Step (ICLR-2024)

Reviewer comment (Experiments):
The experiments were conducted solely on examination datasets. For more robust conclusions, it would be beneficial to conduct experiments on a broader range of significantly diverse datasets.

Author response:
Thank you for the thoughtful comments. We agree that it is worth exploring this method in more general settings. We provide some evidence of generalization beyond math in Table 1 (to chemistry and physics in particular), and we hope that future work will continue to study this further.

Reviewer comment (Experiments):
Do the authors have more results on significantly different datasets?

Author response:
Thank you for the thoughtful comments. We agree that it is worth exploring this method in more general settings. We provide some evidence of generalization beyond math in Table 1 (to chemistry and physics in particular), and we hope that future work will continue to study this further.

Reviewer comment (Novelty):
It is a well-known problem in the RL community that providing dense rewards can be better than providing sparse rewards, and there is a research direction on reward shaping that is very relevant to this idea. Therefore, the idea of providing intermediate rewards for training can lack novelty.

Author response:
You are correct that dense rewards are well studied in RL. However, the majority of LLM RL literature is still focused on sparse rewards. That our methods strongly improve on current baselines shows that reward shaping in this domain is understudied. We think it is important for performance and alignment of LLMs to investigate intermediate rewards further (hence our contribution of a large open-source dataset of dense human feedback).

Reviewer comment (Experiments):
The paper mainly conducts experiments in a math dataset, where it is easy to provide step-by-step intermediate rewards. However, there are many other tasks such as essay writing and story generation where it can be hard to provide such feedback. Therefore, it is unclear if the method is generalizable.

Author response:
We agree with your second point that these methods should be explored on more general distributions. However, the purpose of this paper is to present a clear, compelling argument that researchers should explore process-level feedback further. For this, it is better to start with domains that are easier to evaluate.

Reviewer comment (Experiments):
Under the same budget, is it better to provide step-level rewards on a small set or outcome-level rewards on a large set?

Author response:
To the question: We explicitly argue in section 2.6 that our process-level labels are equivalent in cost to outcome-level labels (so under the same budget the step-level rewards are better), and figure 4 shows about how big a set of outcome-level labels would be needed to recover the performance of our process-level model.

Reviewer comment (Experiments):
The authors haven't applied it to RLHF. It is still not clear how process supervision and outcome supervision affect the generation model performance. Actually, if outcome supervision is large and diverse enough, the model trained with outcome supervision can also do process supervision by feeding the reason path step by step.

Author response:
We appreciate the feedback. We address your concerns below.

2. The primary purpose of this work is to study how to train reliable reward models, which is an important problem in its own right. While we agree that there are interesting connections to RLHF that are worth investigating further, we leave that to future work.


We agree (and argue exactly in section 7.1) that a large enough outcome supervision dataset should recover the performance of our process reward model. I am not certain this is a weakness. We believe this is actually the core contribution of the paper: It is expensive to do RLHF because human feedback is expensive, so we show you can do RLHF more cheaply by collecting more dense labels from humans.

Reviewer comment (Experiments):
Conducting RLHF would be more impressive.

Author response:
We appreciate the feedback. We address your concerns below.

2. The primary purpose of this work is to study how to train reliable reward models, which is an important problem in its own right. While we agree that there are interesting connections to RLHF that are worth investigating further, we leave that to future work.

Reviewer comment (Reproducibility):
The reproducibility for this work is concerning. It is hard to understand under what conditions one can successfully train an effective process reward model. The authors did not have sufficient details for both models and data. In the paper, the authors stated, "The small-scale base models are similar in design to GPT-4, but they were pretrained with roughly 200 times less compute." However, the paper neither reveals the size of the small-scale base models nor the large-scale base models. It is unclear whether the data gets the PRM to work or the scale of the model that gets it to work. I understand that work done at industry has its confidentiality rules to follow, but the authors can maybe conduct experiments on open models on their pretraining and fine-tuning datasets? Due to the reproducibility issue, I think this work would be a better fit for a blog post than a main conference paper.

Author response:
We also appreciate the concerns around reproducibility. While we believe our small-scale results should be wholly reproducible on open-source models, it would have been more responsible of us to test that directly. The large-scale results are likely not replicable on any currently-available open-source models. This is an unfortunate limitation of the current state of the technology and a valid criticism. We still believe it’s important that we report results at the frontiers of scale. We think this is a valid contribution to the literature, since our results suggest that process supervision enables more cost-efficient reward model training at all model sizes.