Paper: ARGS: Alignment as Reward-Guided Search (ICLR-2024)

Reviewer comment (Writing):
There are some serious issues in citation. "Decoupled weight decay regularization" is an ICLR-19 paper, not an arxiv. Please refer to https://openreview.net/forum?id=Bkg6RiCqY7. The authors should list all wrong citations and revise them. I will check the similar issues in all citations one by one.

Author response:
We appreciate your attention to detail. During the manuscript preparation, we made a conscientious effort to adhere to the proceeding format when citing relevant papers. However, we acknowledge the possibility of oversights during the editing phase. As suggested, we conducted a comprehensive check and revised several citations, including the one you highlighted.

It is important to note that some of the cited papers currently exist solely on arXiv without an associated proceeding venue, primarily due to their recent release, especially for works originating in 2023. For these instances, we have retained the citations in their current form and will update them when they are formally published.

Reviewer comment (Experiments):
Qualitative results are limited. I suggest that the authors provide more results to support the claims. It is hard for me to have a clear understanding of the improvements. If an anonymous web demo or a code link is provided, I will revise my rating.

Author response:
In Section 3.3, we conducted qualitative evaluations across 300 randomly sampled prompts from the test set of HH-RLHF. Table 2 presents the qualitative evaluation results, measured by the percentage of win-ties of our method over the alternative decoding strategies. During rebuttal, we have updated our draft by incorporating more qualitative examples. For details, we direct the reviewer to **Appendix D**.

To facilitate further scrutiny of our work, we have included an anonymous link to our code (in the global response to all reviewers). This link will provide you with access to the experiments and implementation details.

Reviewer comment (Presentation):
In Figure 2, different types of lines are not shown in the figure. It is not very clear.

Author response:
That's a great catch. For clarity, we have revised the figure by changing the legend (ARGS-greedy 10 and 40) to a rectangle instead of a line.

Reviewer comment (Theory):
Compared with classical decoding methods, ARGS has higher time complexity. Although the $k$ can be small, it also has a higher complexity. Besides, small $k$ is not good for performance.

Author response:
We would like to point out that employing small $k$ yields similar performance as large $k$. This has been empirically validated in **Section 3.2**. Specifically, Figure 2 illustrates that the performance variation between $k=40$ (darker color bars) and $k=10$ (light color bars) is slight, suggesting that a large number of candidates may not be essential for producing aligned generations.

Empirically, we find that the time complexity overhead can be considerably small. For example, using the OPT-2.7b base model, the generation time per response increases only by 1.9 times when comparing ARGS ($k=10$) with conventional greedy decoding. Despite the slight increase in processing time, there was a notable enhancement in reward performance by $\uparrow$ 6.8%, demonstrating the existence of a reasonable tradeoff between reward optimization and inference speed. The gap can be further reduced by employing a smaller reward model, or parallelizing the reward computation across $k$ candidates. 

Our discussion on time complexity has also been endorsed by reviewer Mqii, who commented: 
> _"The authors do discuss the extra computation added at inference time and show its feasibility. I believe that **such a method is interesting and useful for the literature even with this extra weight at inference**. For example, it can be used to iterate over different reward models before running only one finetuning, or used directly if we have a small enough and good reward model."_

Reviewer comment (Experiments):
The latest baseline method is a paper published in 2022. More baselines should be compared.

Author response:
We have added a latest baseline DPO (Direct Preference Optimization) [1] from NeurIPS 2023. The results have been added to **Section 4**.

Reviewer comment (Novelty):
I am concerned about the technical novelty of the paper. The idea of reward-guided search has been proposed in SCST (Self-Critical Sequence Training). Besides, the technical contribution compared with the baseline is a tricky implementation, which is marginal.

Author response:
We thank you for pointing out the literature [4]. Upon close examination, the work appears to be different from ours in terms of problem setting, methodology, and various experimental details. 

- Problem setting: While SCST focuses on the problem of image captioning, our distinct focus is the alignment of large language models. To the best of our knowledge, we are the first to introduce a reward-guided search mechanism for the alignment problem. 
- Methodology: SCST does not train the reward model using human preferences like we do, and directly employs greedy decoding in inference time. Instead, our key intellectual contribution is a reward-guided decoding procedure in testing time, which combines explicitly reward scores as well as the output of language models. Moreover, SCST involves an RL algorithm, whereas our framework circumvents the need for RL training. 
- Other empirical distinctions: SCST was developed based on LSTM and CNN architectures, whereas the alignment literature today commonly focuses on transformer-based autoregressive language models. It's also worth noting that the evaluations, experiments, and metrics are all different between the two works.

Reviewer comment (Evaluation):
The user study is needed for evaluation.

Author response:
We employ a GPT-4-based evaluation approach to assess the quality of responses. Research, as discussed in [2], has shown that using a GPT-4 proxy aligns with human evaluations in over 80% of cases, providing a scalable method to approximate human preferences. This evaluation method is prevalent in recent literature, as evidenced by studies such as [1, 3].

Due to practical considerations, our Institutional Review Board (IRB) protocol for conducting human evaluations is currently undergoing review at our institution. We fully intend to incorporate human evaluations into our study as soon as our IRB protocol receives approval.

Reviewer comment (Evaluation):
Missing discussion on limitation.

Author response:
As suggested, such a discussion has been added to our revised manuscript. Please see **Section 6**.

Reviewer comment (Experiments):
Limited evaluation: The evaluation of the ARGS framework is limited to a few specific tasks (e.g., harmfulness), and it is unclear how well the framework would perform on other tasks (more complex ones like multi-step reasoning), especially when a good reward model is not easy to train. This may limit its applicability in certain domains.

Author response:
> **Evaluation on other tasks**

You raised an excellent point. For our current evaluations, we follow the standard and commonly used benchmarks in alignment literature. In particular, HH-RLHF from Anthropic and Stanford Human Preferences (SHP) are among the largest and publicly available datasets for alignment research. These tasks allow us to draw comparisons with existing approaches more easily and directly. As the first study to introduce the decoding-time alignment approach, we believe it's important to understand the potential of our approach more deeply in the existing realm of alignment tasks, before we expand to other tasks. 

With that being said, we do agree that evaluating on more complex tasks such as multi-step reasoning would be valuable. During rebuttal, we looked further into this, however, we could not locate large-scale human preference datasets on multi-step reasoning, which hinders the feasibility of training the reward model. We have revised our draft to acknowledge this limitation and remain interested in delving deeper into more complex tasks as part of our future work.

Reviewer comment (Evaluation):
Unfair evaluation: The evaluation of the ARGS framework is evaluated on the score from reward model. However, there are some limitations: (1) ARGS will certainly achieve higher scores since the RM is integrated during the decoding process. Essentially, applying any RM constraints during the decoding stage will result in higher scores when being evaluated by that RM. (2) The calibration of RM remains unclear -- does a higher reward score certainly lead to a better response, especially the reward score difference is less than 1? Overall, I think it not very fair if the authors use the same RM in their methods and evaluation.

Author response:
> **Evaluation metrics**

We understand your concern out of the Goodhart's Law :) We share the same and thus have taken the following additional steps to ensure our evaluations are comprehensive:

- We performed an evaluation where the evaluating reward model is _different_ from the one we used in training. In Section 3.4, we conducted experiments where we employ fined-tuned OPT-1.3b and OPT-2.7b for decoding, and OPT-125m as reward models. We evaluate the models on random 1,000 samples of the test set from the Stanford Human Preferences (SHP) dataset, and the average reward is calculated by the OPT-350m reward model. As shown in Figure 3, ARGS consistently outperforms the greedy baseline.
- To address the nuanced aspects of language quality that the standard metrics (diversity, coherence, reward) may not comprehensively capture, we also adopt a GPT-4-based evaluation approach for comparing the quality of responses. Table 2 presents the GPT-4 evaluation results, measured by the percentage of win-ties of our method over the alternative decoding strategies. A higher percentage indicates that our proposed method is more proficient in generating responses that exhibit not only contextual relevance and accuracy but also helpfulness and harmlessness. This observation is consistent with the outcomes of the reward-based evaluation discussed in Section 3.2.

Reviewer comment (Experiments):
A good reward model is vital in ARGS, have you ever tried to enhance the RM? The paper does not show the relation between RM quality and Effectiveness of ARGS. For example, you might use some techniques from (a) The Trickle-down Impact of Reward (In-)consistency on RLHF 2023, (b) Aligning Language Models with Preferences through f-divergence Minimization 2023, (c) Fine-Grained Human Feedback Gives Better Rewards for Language Model Training 2023 to strengthen your RM.

Author response:
> **Relation between RM quality and effectiveness of ARGS**

Another great point raised! 

Indeed, our experimental results in Section 3.4 reveal such a connection between RM quality and the effectiveness of ARGS. In particular, the RM quality can be modulated by the model capacity. We experimented with a smaller capacity model OPT-125M, along with a larger capacity model OPT-350M. The reward modeling accuracy, as well as the ARGS performance (average reward, diversity, coherence), is summarized in the table below. We observe that a higher RM accuracy in general leads to stronger decoding performance by ARGS.

**Base model**: OPT-1.3b

| Reward Model | Evaluation Accuracy (%) | Average Reward | Diversity | Coherence |
| ------------ | ----------------------- | -------------- | --------- | --------- |
| OPT-125m     | 52.62                   | 5.698          | **0.322** | 0.376     |
| OPT-350m     | **53.16**               | **5.979**      | **0.322** | **0.389** |

**Base model**: OPT-2.7b

| Reward Model | Evaluation Accuracy (%) | Average Reward | Diversity | Coherence |
| ------------ | ----------------------- | -------------- | --------- | --------- |
| OPT-125m     | 52.62                   | 5.71           | 0.369     | 0.431     |
| OPT-350m     | **53.16**               | **5.929**      | **0.380** | **0.435** |

This experiment also informs us about a meaningful future direction to explore ARGS decoding from an RM modeling perspective. The papers you recommended are excellent starting points for this. We hypothesize that coupling advanced RM modeling objectives (beyond pairwise ranking loss) with ARGS can help further enhance the generation quality. We have added those discussions in our updated manuscript accordingly.

Reviewer comment (Evaluation):
How would you evaluate the instruction-following ability of model based on ARGS? Since Diversity/Coherence are measuring the naturalness of model responses, there is a blank space for quantifying whether model completes the instruction (in your case, it is ethical property).

Author response:
> **Instruction following evaluation**

We evaluate the instruction following ability based on GPT-4 evaluation (Section 3.3). We explicitly instruct GPT-4 to assign the score to the responses based on helpfulness, harmlessness, relevance, accuracy, and insightfulness (see the prompt template attached in **Appendix B**). 

Table 2 presents the GPT-4 evaluation results, measured by the percentage of win-ties of our method over the alternative decoding strategies. A higher percentage indicates that our proposed method is more proficient in generating responses that exhibit not only contextual relevance and accuracy but also helpfulness and harmlessness.

Reviewer comment (Evaluation):
The main question I am interested in is: What is the performance loss from using the reward function on incomplete prefixes?

Author response:
> **Further analysis of using reward model on prefixes**

You raise a very insightful question. To better understand this, we analyzed the average reward w.r.t. different locations. Specifically, for a given $\texttt{prompt}$, we denote the sequence of generated tokens as $x_1$, $x_2$,.., $x_t$, and so on. The average reward at $t$-th position is calculated as $r([\texttt{prompt},x_{\le t}])$, averaged over the entire test set of HH-RLHF. We use the SFT-ed Llama-7B model for decoding, and report the statistics below for your reference.

| Predicted token index $t$ | Average Reward |
| ------------------------- | -------------- |
| 10th                      | 5.38           |
| 50th                      | 6.42           |
| 100th                     | 6.80           |


As ARGS predicts more tokens, it can be seen that the average reward monotonically increases. This indicates that using the reward model on partial prefixes can steer the search toward generating sequences with higher overall reward scores.

Reviewer comment (Evaluation):
When is the predicted reward from the reward function most unreliable (likely on sequences further from completion)?

Author response:
> **Further analysis of using reward model on prefixes**

You raise a very insightful question. To better understand this, we analyzed the average reward w.r.t. different locations. Specifically, for a given $\texttt{prompt}$, we denote the sequence of generated tokens as $x_1$, $x_2$,.., $x_t$, and so on. The average reward at $t$-th position is calculated as $r([\texttt{prompt},x_{\le t}])$, averaged over the entire test set of HH-RLHF. We use the SFT-ed Llama-7B model for decoding, and report the statistics below for your reference.

| Predicted token index $t$ | Average Reward |
| ------------------------- | -------------- |
| 10th                      | 5.38           |
| 50th                      | 6.42           |
| 100th                     | 6.80           |


As ARGS predicts more tokens, it can be seen that the average reward monotonically increases. This indicates that using the reward model on partial prefixes can steer the search toward generating sequences with higher overall reward scores.

Reviewer comment (Experiments):
Comparisons against DPO [1] should be included, even if it could be considered concurrent work.

Author response:
> **Comparison with DPO**

As suggested, we additionally conduct comparisons with the latest approach DPO [1] and report the result in the table below. The comparison and full training configurations have also been added to our manuscript (see **Section 4** and **Table 8**). Our method overall achieves a higher average reward. During experimentation, we notice the tendency for DPO to generate sometimes repetitive words, which potentially leads to low diversity. In contrast, our method more stably generates diverse responses. 

| Method | Category       | Average Reward | Diversity | Coherence |
| ------ | -------------- | -------------- | --------- | --------- |
| ARGS   | Decoding-based | 5.98           | 0.322     | 0.390     |
| PPO    | Training-based | 5.88           | 0.247     | 0.264     |
| DPO    | Training-based | 5.65 | 0.096 | 0.372  |

[1] Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS 2023.

Reviewer comment (Writing):
Can you add a sentence in section 3.1 stating that SFT on HH-RLHF means fine-tuning on the winning responses, if that is what was done.

Author response:
> **Writing suggestions** 

All fixed - the changes have been marked in red in our updated manuscript. Thank you for the careful read!

Reviewer comment (Evaluation):
In section 3.2, can you say a relative improvement of 19.56% *in average rewards*.

Author response:
> **Writing suggestions** 

All fixed - the changes have been marked in red in our updated manuscript. Thank you for the careful read!

Reviewer comment (Experiments):
You could combine multiple reward functions pretty easily.

Author response:
> **Combining multiple reward functions**

That's a really interesting suggestion. As you recognized, our approach indeed offers the flexibility to integrate multiple reward signals in decoding-time, without having to re-train the PPO model. In our current exploration, we focus on using a single reward function, which allows us to draw comparisons with existing approaches more easily and directly. As the first study to introduce a decoding-time alignment approach, we believe it's important to understand the potential of our approach more deeply in the existing realm of alignment tasks, before we expand to more complex settings. 

With that being said, we do agree that evaluating more complex tasks, such as multi-step reasoning, would be valuable. To do so, one needs more thoughtful and meaningful construction of the tasks, as well as the availability of diverse human preference datasets to train different reward functions. For this reason, we would like to spend more time diving deeper into this aspect in our future work, and hopefully present more conclusive findings after.

Reviewer comment (Experiments):
It would be interesting to see if other reward models return sensible rewards on prefixes.

Author response:
Thank you for taking the time to read our response and increasing the score! We agree the suggested experiment could be intriguing, and will include this in our future version.