Paper: Chain of Hindsight aligns Language Models with Feedback (ICLR-2024)

Reviewer comment (Experiments):
The largest model tested is GPT-J, which is a 6B model. It would be great that larger and more latest LLMs can be also tried, such as LLAMA-2.

Author response:
**#1: Regarding your suggestion on larger and more recent LLMs**

We have conducted experiments using the latest 7B LLaMA model. In our study, we compared SFT, RLHF, and CoH against Koala, a state-of-the-art LLaMA-based chatbot, serving as a robust calibration baseline. Our comparison (see Figure 9) of these methods applied to LLaMA on HH and summarization datasets with Koala used pairwise human evaluation. Notably, Koala is developed from the ShareGPT dataset, known for its superior quality and greater diversity compared to open-source datasets like HH and summarization. The results indicate that our CoH approach performs on par with Koala, and when combined (CoH + Koala), it slightly outperforms Koala based on human ratings. However, both C-SFT (conditional SFT) and SFT significantly lag behind Koala, highlighting the effectiveness of CoH in leveraging human preferences for learning.

Reviewer comment (Experiments):
In page 4 for the "Training" section, it is mentioned that "the model can simply ‘copy’ the example without learning to understand the underlying task". To circumvent this, this work proposes to "randomly mask between 0% and 5% of past tokens during training". What do the "past tokens" refer to? How is this masking ratio determined? Have you done some tuning on this hyper-parameter?

Author response:
**#2: Clarification on "past tokens" and masking ratio**

By “past tokens,” we refer to the causal tokens visible to a given token. Our approach randomly masks past tokens to prevent overfitting during training. The masking ratio was determined through preliminary experiments; we found that a larger ratio hindered training efficiency, while a smaller one made negligible difference.

Reviewer comment (Reproducibility):
For the above point 2, this work proposes to "added a regularization term which maximize the log likelihood of the pretraining dataset". Could you specify how much pretraining data you need to use for such regularization? What is the added overhead to the training process?

Author response:
**#3: On regularization and pretraining data**

For the regularization, we used approximately the same number of tokens as the human feedback data. This approach resulted in about a 40% increase in wall clock time for training. We will clarify this further in our revision.

Reviewer comment (Evaluation):
For the dialogue task, it would be great that besides the human evaluation, another automatic evaluation based on GPT-4 can be also provided. This work was first out in this Feb or Mar., at that time, we have tried replicating this method using GPT-J and Anthropic-HHH dataset and this provided code base by the authors. We have used the same setting except that we were not using the regularization term, but we have found that the resulted model tends to generate long responses with weird grammar and repetitions. That is, the generation quality is pretty poor.

Author response:
**#4: Suggestion for GPT-4 based evaluation**

While we primarily focused on human evaluation for its reliability, we acknowledge its limitations in terms of scalability and expense. We have conducted a GPT-4 evaluation for LLaMA-trained SFT, PPO-RLHF, and CoH, following the MT-Bench evaluation framework. Our results show that CoH achieves the highest score, significantly outperforming SFT and PPO. We will include results of GPT-4 based evaluation for more tasks in the revision. 

| Model | Turn     | Score |
|-------|----------|-------|
| SFT   | 1        | 5.34  |
| PPO   | 1        | 5.82  |
| CoH   | 1        | 6.03  |
| Model | Turn     | Score |
| SFT   | 2        | 4.23  |
| PPO   | 2        | 4.56  |
| CoH   | 2        | 4.87  |
| Model | Average  | Score |
| SFT   |          | 4.79  |
| PPO   |          | 5.19  |
| CoH   |          | 5.45  |

We appreciate your efforts in using our open-source code. The issue you encountered with model generation quality could be due to the absence of regularization, leading to overfitting, or incorrect hyperparameter settings. We plan to enhance our codebase with more detailed instructions and example scripts to aid future research and ease of use.

Reviewer comment (Reproducibility):
During inference, could you control the model to just generate the good response or the model would generate both good and bad responses and extra post-processing is needed? I doubt the case is the latter one based on my intuition and our replication experiments, which would make the inference latency double compared with other RLHF methods.

Author response:
**#5: Does inference process need post-processing**
No extra post-processing is required during inference. The model is designed to generate quality responses directly. However, we are exploring potential post-processing methods that might include using search algorithms to select the best outputs or combining various token outputs for exploration. These are promising directions for future research.

Reviewer comment (Experiments):
More details needed about RLHF. At present, it's unclear why CoH outperforms RLHF. Is the trained RM ineffective at modeling preference (add RM performance to Fig3)? Or is the learning algorithm unable to leverage the RM effectively (could it be the choice of prompts used for RLHF?), in which case an alternate RM-based baseline could be considered (e.g. rejection sampling, reinforced self-training)?

Author response:
**#1: More discussion on RLHF**

PPO-based RLHF suffers from reward over-optimization, as indicated in [1]. This paper also suggests training a very large model to address this issue. Despite experimenting with large models (up to 13B parameters due to compute constraints) for our PPO RLHF, we observed no significant improvements. Our hypothesis is that an even larger-scale reward model might be necessary. Additionally, we have tuned the PPO pipeline using the HuggingFace TRL library and experimented with varying the pretraining data regularization term for PPO RLHF training. Due to the complexity of reward learning and PPO RL training with multiple models, achieving optimal performance is challenging. CoH's advantage lies in its likelihood-based training, which aligns with the pretraining objective, thereby facilitating tuning and ensuring effective performance. We anticipate that future research will further explore the strengths and weaknesses of PPO RLHF and CoH, potentially leading to improved training algorithms.

[1] Scaling Laws for Reward Model Overoptimization by Leo Gao, John Schulman, Jacob Hilton

Reviewer comment (Experiments):
Why do you only prompt with 'Good:' at inference time? It seems that an advantage of the different natural language feedback templates is that the CoH-trained LM will (hopefully) learn to *consider an initially generated bad response, and leverage it to produce a good response*. Perhaps leveraging this capability, and giving the LM an attempt to produce a better response, may yield further benefits. Some discussion or analysis about the potential of different natural language feedback templates at inference time would be useful.

Author response:
**#2: On using 'Good:' prompts at inference time**

We agree that prompting CoH-trained LM with other instructions at inference time could improve performance. To this end, we have conducted experiments to compare prompting model with `Good:`, and prompting model with `Good:` followed by another round of `Better:`. The results shown in Figure 8 shows that CoH-trained LM can learn to consider the initially generated response and improve upon it.

Reviewer comment (Evaluation):
More discussion is warranted about the performance relative to C-SFT. If my understanding is correct that C-SFT is essentially training with ("good: {good response}", "bad: {bad response}"), then I wonder if CoH with only ("good: {good response} bad: {bad response}", "bad: {bad response} good: {good response}" is expected to perform better (under the current inference setup)? If yes, why is this the case? If not, where is the advantage of CoH relative to C-SFT coming from?

Author response:
**#3: More discussions about the performance relative to C-SFT.**

CoH differs from C-SFT in that it allows the model to reference a bad response when predicting a good one, and vice versa. This approach is crucial as it enables the model to learn directly from feedback and examples. As shown in experiments, CoH outperforms C-SFT substantially.

Reviewer comment (Writing):
I'm not certain I understood this properly: Does 'CoH w/o lang' mean that the template will always be something like "Good: {good} Bad {bad}"? The text mentions "fine-grained language feedback" and it's not whether this is referring to the feedback templates or something else.

Author response:
**#4: Clarification on 'CoH w/o lang'**

By this, we mean consistently training with the "Good: {good} Bad: {bad}" template. The term “fine-grained language feedback” refers to the use of a variety of task-dependent language templates, such as "A good summary: {good}, a worse summary: {bad}" and "You are a helpful assistant: {good}, you are an unhelpful assistant: {bad}". These templates are detailed in the box at the bottom of page 3 and in Table 6. We will clarify this in the revised manuscript.

Reviewer comment (Experiments):
Do you have insights on how this method would scale from pairwise comparisons to N-way rankings? I thought that WebGPT had >2 responses per prompt (but might be mistaken) -- if so, it would be great to understand how this was handled.

Author response:
**#5: On scaling to N-way rankings**

CoH can be readily adapted for N-way rankings by structuring N-way ranked outputs into a sequence. However, a potential challenge arises if N is significantly large, as this would proportionally increase the context size. Regarding the WebGPT dataset, each prompt includes two answers, which we rank according to the accompanying scores. We will provide more details about this in our revision.

Reviewer comment (Experiments):
Is the loss masking during CoH training really important? I would not have expected it to make a substantial difference.

Author response:
**#6: Importance of loss masking during CoH training** 

Loss masking is beneficial for stabilizing training, as predicting control tokens can introduce noise into the prediction of desired outputs. It also aids in regularizing the model and preventing overfitting to specific training examples. While CoH is effective without loss masking, we included it as it enhances performance without additional computational costs.

Reviewer comment (Experiments):
Limited Scope of Testing: The paper only considers two evaluation benchmarks such as dialogue and summarization benchmarks. It is not clear how the model performs on the standard academic benchmark. Broader testing across diverse datasets and real-world scenarios would be necessary to fully validate the approach.

Author response:
We acknowledge that broader scope of testing as an area for future improvement. Our initial focus was on demonstrating the effectiveness of CoH in tasks where feedback dynamics are crucial, such as dialogue and summarization, and evaluated on the primary dataset in each category. However, we agree that broader testing across diverse datasets is essential for validating the method's applicability in a wider context. In future work, we aim to include a more varied set of benchmarks, including code generation from unit tests feedback, to better evaluate the method's generalizability and performance in different scenarios.

Reviewer comment (Novelty):
Some important studies [1,2] are missing. It would be better to include them and have a discussion.

[1] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Author response:
**#2: Include a discussion on related studies**

We appreciate your suggestion to include the studies “RAFT” and “Direct Preference Optimization”. These are indeed significant contributions to the field. We will incorporate these references into our paper and discuss how our work aligns with, and differs from, these studies.