Paper: RAIN: Your Language Models Can Align Themselves without Finetuning (ICLR-2024)

Reviewer comment (Experiments):
Baselines: The paper does not provide a detailed comparison of RAIN with other methods that align pre-trained language models using RLHF or instruction tuning.

Author response:
Indeed, we have conducted such comparative experiments. As detailed in Figure 4 on page 8 of our paper, our comparison includes RLHF and RLAIF, both of which involve SFT (akin to instruction tuning) and RL phases. The results from these comparisons show that while RLHF and RLAIF require additional data, RAIN demonstrates improved performance.

Furthermore, we have tested RAIN on models that were aligned using various methods. These experiments are presented in Table 1 on page 7, and in Table 2 on page 8. The results from these tables indicate that RAIN is capable of further enhancing the safety of models that have already been aligned using existing methods.

Reviewer comment (Experiments):
Limited evaluation on real-world scenarios: The experiments in the paper are conducted on benchmark datasets and may not fully capture the complexity and diversity of real-world scenarios. This means that the effectiveness of RAIN in real-world applications may be different from what is reported in the paper.

Author response:
Thank you for your feedback. We have conducted human evaluations, which closely resemble assessments in real-world scenarios. The results, presented in Table 8 on page 9 of our paper, clearly show that RAIN also performs effectively under human evaluation. This approach not only validates the efficacy of RAIN in a controlled experimental setup but also demonstrates its practical applicability in real-world contexts.

Reviewer comment (Experiments):
How does RAIN compare to other methods that align pre-trained language models using reinforcement learning or instruction tuning?

Author response:
Indeed, we have conducted such comparative experiments. As detailed in Figure 4 on page 8 of our paper, our comparison includes RLHF and RLAIF, both of which involve SFT (akin to instruction tuning) and RL phases. The results from these comparisons show that while RLHF and RLAIF require additional data, RAIN demonstrates improved performance.

Furthermore, we have tested RAIN on models that were aligned using various methods. These experiments are presented in Table 1 on page 7, and in Table 2 on page 8. The results from these tables indicate that RAIN is capable of further enhancing the safety of models that have already been aligned using existing methods.

Reviewer comment (Experiments):
How scalable is RAIN to larger models or datasets, and what are the computational requirements for using RAIN on such datasets?

Author response:
One of the key advantages of RAIN is that it does not require additional memory overhead, which significantly aids its scalability to larger models. To demonstrate this, we have conducted extensive experiments with models as large as 70B parameters, which is the upper limit in the LLaMA series available to us. The results presented in Figure 1 on page 2 and Figure 3 on page 7, along with Table 1 on page 7 in our paper, clearly illustrate that the effectiveness of RAIN tends to increase with larger models. Furthermore, Table 7 on page 9 demonstrates that the relative additional time overhead required by RAIN diminishes as the size of the model increases. Additionally, RAIN is a universal, data-agnostic, and training-data-free algorithm, making it inherently scalable to large test datasets.

Reviewer comment (Experiments):
The proposed inference-time method has high computational cost. This will make it challenging for the method to be deployed in practice. (On the other hand, one can perform fine-tuning on generated samples so that the sampling approach need not be performed at inference time; the authors didn't not explore this approach.)

Author response:
In the conclusion section, we discuss the limitations of RAIN: Compared to standard auto-regressive inference, RAIN requires a longer, yet acceptable, inference time. A potential solution to this issue could involve using RAIN to generate data and then fine-tuning models with this data. However, fine-tuning LLMs is resource-intensive and contradicts the aim of our paper, which is to "investigate the feasibility of enabling (self-)alignment in fixed LLMs without fine-tuning." Therefore, we have deferred this approach to future work.

Reviewer comment (Reproducibility):
The proposed method is complicated.

Author response:
Intuitively, RAIN conducts Monte-Carlo search on the tree consisting of token sets and dynamically reduces the weight of harmful token sets, with backward rewind and forward generation steps until the output content is self-evaluated as harmless. We also modify the Monte-Carlo tree search to adapt to the LLM alignment tasks by introducing the token similarity check. The potentially "complicated" design is to improve the search efficiency.

Reviewer comment (Experiments):
**Limited Applications**: The proposed nested iterative search algorithm significantly increases the computational overhead at inference time (>3.7x on 70B models, and >4.3x on 30B models), making it intractable for practical purposes as the running time for these large models are already very large. In the meanwhile, the tree-search operation makes it extremely hard to perform parallelized optimization and can induce significant I/O throughputs as many operations in this proposed algorithm may not be able to easily run on GPU, so batch inference will be much harder to implement, further increasing the real latency on the user side. Considering this significant computational overhead, it is not clear whether this performance gain reported in this paper is worth it -- maybe practitioners should instead sample many more outputs via some diversity-enforced sampling method [1] (parallelizable and has been largely optimized via various open-source libraries such as VLLM [2]), and apply a well-tuned reward model to pick up the best outputs (note that this best-of-K/rejection sampling method is not even listed as baseline in this work).

Author response:
> Q1. Best-of-K/rejection sampling method is not listed as baseline in this work.

In fact, we have conducted a comparison with the method you mentioned. This is detailed in Table 5 on page 9 of our paper, where the  best-of-K method is referred to as "Vanilla (Repeated)." Best-of-K is a specific case of rejection sampling, and if the evaluation is effective, best-of-K should not perform worse than rejection sampling. The experimental results demonstrate that even with 500 trials (best-of-500), for the 7B model, best-of-K barely improves the model’s safety. With the 30B model, RAIN shows a significantly enhanced performance. This clearly indicates the superior efficacy of RAIN compared to the best-of-K/rejection sampling approach.

Reviewer comment (Novelty):
**Partially-Justified Motivation, and perhaps Marginal Contributions**: The author argues that "Beyond performance, the primary goal of this paper is to investigate the feasibility of enabling self-alignment in fixed LLMs without fine-tuning or RL" (in Introduction). However, it is not clear the necessity of applying the proposed iterative tree-search algorithm to verify this feasibility. As the authors recognize in the paper, there are already several tree-search-style inference-time algorithms in adjusting LLM outputs (the author mentioned [3] in the paper, and I think [4] is also related), and the self-evaluation does not seem like a new contribution in decoding content planning to improve both LLM reasoning and alignment (e.g., [1,5,6]) without learning. Though these works may not run on the specific task set as the author, they do have the potential to be strong competitors and some of them have already achieved the verification that a "learning-free algorithm can improve alignment via self-evaluation".

Author response:
> Q2. Even though other related works have not been applied to the specific task set used by the author, they still have the potential to be strong competitors.

For [1] (RLAIF), we have included comparative results in Figure 4 on page 8. The method described in [6] is applied to models trained with RLHF, which diverges from our focus on non-finetuned models. To the best of our knowledge, for the methods in [3,4,5], there is no clear evidence suggesting their applicability in the domain of LLM safety. The method in [5] necessitates interaction with the environment to gather feedback, which, in our research context, would mean requiring real user feedback. This is impractical in the field of safety and makes a comparison with RAIN unfair. The problems addressed by RAIN and those tackled by the aforementioned studies are fundamentally different. RAIN's external form is consistent with that of auto-regressive generation, which makes it adaptable to a wide range of language generation tasks. The studies in [3,4] are primarily concerned with reasoning tasks, which inherently allow for explicit planning. For example, [3] investigates tasks like the 24-point game, which involves multi-step number combinations, while [4] focuses on logical reasoning, which can be broken down into multi-step selection and deduction processes. Such explicit planning structures are absent in tasks like harmless generation, leading to different search spaces. The search space in [3,4] revolves around actions like combining numbers, whereas RAIN's search space is broader, encompassing the entire language space.

Despite various challenges, we made our utmost effort to adapt LEAP proposed in [4] for the task of harmless generation and subsequently conducted experiments on the HH dataset. Here are the experimental details and results. We replaced the actions in LEAP with the selection of token sets. LEAP uses $u=\log p_{\text {sel }}(\mathbf{s})+\alpha \Delta u$ and $\Delta u = \max_d \log p_{\text{ver}}\left(x_0 \mid y_d\right)$ to determine the score, where $p_{\text {sel}}$ is given by the selection model and $p_{\text{ver}}$ by a different verification model. As we did not train a verification model, we set $p_{\text {sel}}$ to be the probability obtained from querying LLMs and $p_{\text{ver}}$ as the normalized self-evaluation score, ensuring the meanings of both terms remain unchanged. The experiment's hyper-parameters were consistent with the original paper of [4], such as $B=5$, $\alpha=10$, etc. The results are as follows:

|      Method       | Vanilla |  LEAP  | RAIN  |
|:-----------------:|:-------:|:------:|:-----:|
| harmlessness rate |  82.2%  | 89.1%  | 97.0% |

Our findings indicate that while LEAP improves harmlessness over vanilla auto-regressive generation, RAIN significantly outperforms both methods.

Reviewer comment (Experiments):
**Weak Baselines**: As explained in the above points, best-of-K/reject sampling, and existing tree-search algorithm (without iterations and back-tracking. Introducing self-evaluation to them does not seem too hard -- correct me if I am wrong.) have full potential to be the strong competitors to the proposed method (some of them may have already been deployed somehow by companies), given sufficient computational budgets (perhaps 3-4x more than normal inference time, to make a fair comparison to the proposed algorithm). None of them is made as a baseline in this paper and the author mainly compares with vanilla auto-regressive generation, which is relatively weak. Note, this is not to say the author needs to achieve state-of-the-art when comparing with these stronger baselines -- this is to help users understand the necessity of deploying the proposed algorithm and choose based on their needs.

Author response:
> Q1. Regarding best-of-K sampling as a baseline: I strongly suggest you have a section of baseline in your formal version explaining all baselines you plan to use for clarification. Currently, there is no clear explanation for it to be regarded as a best-of-K sampling baseline. First, there is no explanation of your sampling parameters (the temperature, the top_p and top_k) -- this is very important as we need to select from a diverse set of generations. Second, why do you think using LLM itself as an evaluator is a valid best-of-k sampling? I think a valid best-of-K sampling baseline should be at least using one of these models (e.g., OpenAssistant Reward Model: https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large).

We appreciate your valuable suggestion and will include a section of baselines in the final version of our manuscript for clearer understanding. In our "Vanilla (Repeated)" approach, we utilized direct sampling from the LLM's original distribution, i.e., setting the temperature parameter to 1, without employing additional sampling strategies such as top_k and top_p. We also conducted additional experiments with different sampling parameters, including those involving a preference model. The results of these experiments are presented in this response.

Using LLM itself as an evaluator may not be the optimal choice for best-of-k sampling. However, under the premise of ensuring comparability and fairness in our experimental setup, we consider it a valid approach. The primary aim of our research is to investigate the feasibility of enabling (self-)alignment in fixed LLMs, rather than solely focusing on performance. Consequently, we restricted RAIN from accessing preference models trained with external knowledge. To ensure a fair comparison, "Vanilla (Repeated)" was evaluated using the same self-evaluation method as RAIN, rather than an external preference model.

Additionally, we have conducted experiments using the preference model you mentioned (OpenAssistant Reward Model: https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large). In these experiments, we scored the generations with the preference model and selected the highest-scoring outputs as the final results. The findings from these experiments, conducted using the LLaMA 30B model on the HH dataset, are as follows:

|      Setting      | temperature=1, top_p=NA, top_k=NA | temperature=0.5, top_p=NA, top_k=NA | temperature=1, top_p=0.9, top_k=NA | temperature=1, top_p=NA, top_k=60 | RAIN |
|:-----------------:|:---------------------------------:|:-----------------------------------:|:---------------------------------:|:-------------------------------:|:----:|
|   self-evaluation |              88.1%               |                86.6%                |              87.6%               |              88.5%              | 97.0%|
| preference model  |              92.5%               |                91.1%                |              92.1%               |              92.1%              |  NA  |

The experimental results show that while there are differences in outcomes due to various sampling strategies, these differences are not substantial. The experimental results also indicate that employing a preference model yields better outcomes compared to self-evaluation. This disparity could be attributed to the model bias inherent in self-evaluation. Notably, despite this, RAIN, which utilizes self-evaluation, still outperforms the best-of-K method that uses a preference model.

Reviewer comment (Reproducibility):
**Unclear Experiment Details**: What are the maximum number of iterations $T$ and the minimum number of iterations $T_m$ used in the experiment? How many tokens are generated for each node? Moreover, how does the author decide each hyperparameter for each task? I checked both the main text and appendix and I did not find how the author set these parameters and based on what observations. These parameters (especially $T$ and $T_m$)  can be very important for both reproduction purposes and for users to see a trade-off between efficiency and performance if they want to deploy the proposed method.

Author response:
> Q3. Unclear experiment details. What are the maximum number of iterations $T$ and the minimum number of iterations $T_m$ used in the experiment? How many tokens are generated for each node?

Thank you for pointing it out. In RAIN, hyper-parameters $T_m$ and $V$ control the early exit strategy, balancing time cost against performance. These parameters are tuned based on the accuracy of self-evaluation: the more precise the self-evaluation, the lower we can afford to set $T_m$ and $V$. For all tasks except truthfulness, we set $T_m=2$ and $V=0.7$, whereas for truthfulness, due to its increased complexity, we used $T_m=16$ and $V=0.8$. Across all tasks, the upper limit of the inner loop iterations, represented by $T$, was fixed at 50. When generating responses, tokens positioned earlier in the sequence have a greater influence on the entire token sequence. Therefore, we set a varying number of tokens for nodes at different positions, specifically in the sequence of 1, 2, 4, 10 (with all subsequent nodes containing 10 tokens each).

Reviewer comment (Writing):
**Hard-to-Read Formulation**: It is very hard to read the formulation presented in Section 3. For example, should not the `node` use different notations from the actual `token`? As each node contains more `attributes` (visit counts, embeddings) than a token? In many places, the authors use the same symbol to represent tokens and nodes. Also, $X$ and $Y$ sometimes seem interchangeable (e.g., $Y_{1:i-1}$ and $X_{1:i-1}$ as the "preceding context"). The usage of the subscript is not careful -- for example, why $S(Y)$ represents the score used to update $Y^*_{a:b}$ without having $a,b$ as input to $S()$?

Author response:
> Q4. Should not the node use different notations from the actual token?

Thank you for your insightful suggestion. In our paper, nodes encompass more than just the token set, including four additional attributes: embedding, probability, visit count, and value. Acknowledging this complexity, it would indeed be clearer to differentiate between "node" and "token set" with distinct notations, enhancing the clarity and precision in our method section.

> Q5. $X$ and $Y$ sometimes seem interchangeable.

As mentioned in the notation section, uppercase letters like $X$ and $Y$ represent sequences of tokens (token set). We use $X$ and $Y$ simultaneously in some places to denote two distinct token sets, such as those corresponding to child and parent nodes.

> Q6. Why does $S(Y)$ represent the score used to update $Y_{a:b}^{*}$ without having $a,b$ as input to $S()$?

In our method, the formula for updating is as follows:
\begin{equation*}
v(Y_{a:b}; Y_{1:a-1})=\frac{1}{\left| {X: Y_{1:b}=\operatorname{prefix}(X)} \right|}\sum_{{X: Y_{1:b}=\operatorname{prefix}(X)}} s(X).
\end{equation*}
This formula indicates that we update $Y_{a:b}$ using $s(X)$. While $a$ and $b$ are not explicit inputs to $S()$, they are implicitly linked through the condition $X: Y_{1:b}=\operatorname{prefix}(X)$. Here, $Y_{1:b}$ serves as a prefix of $X$, and $a$ can range from 1 to $(b-1)$.
The reason for this update is that token sets which are prefixes of $X$ contribute to $s(X)$, and $s(X)$ should update all token sets that contribute to it. We also provide an example in the paper: the value for "Robbing is" should be the average score of "Robbing is illegal" and "Robbing is a serious offense."

Reviewer comment (Presentation):
Algorithm 1 is not referenced in the main text, although it is presented in the methodology section. It would be helpful to mention Algorithm 1 and provide a brief explanation or reference to it where relevant.

Author response:
Thank you for highlighting the oversight regarding Algorithm 1. In the revised version of our paper, we will ensure that Algorithm 1 is clearly referenced. Furthermore, in the methodology section, we will add correspondences between the text and the pseudocode of the algorithm to enhance clarity and understanding.

Reviewer comment (Reproducibility):
In the evaluation and searching process, there are several thresholds mentioned, such as $T_m$ and $V$. It would be beneficial to clarify whether these thresholds have common settings or if they are benchmark-specific.

Author response:
Thank you for your insightful suggestion. In our methodology, we utilize thresholds $T_m$ and $V$ to effectively manage the early exit mechanism, aiming to balance time efficiency with performance. The calibration of these parameters depends heavily on the accuracy of the model's self-evaluation. Generally, the more accurate the self-evaluation, the lower we can set these thresholds.

In our experiments, we identified the truthfulness task as particularly challenging for self-evaluation, requiring different threshold settings. Specifically, we used $T_m=2$ and $V=0.7$ for all tasks except truthfulness. For the truthfulness task, the parameters were adjusted to $T_m = 16$ and $V = 0.8$ to accommodate its unique challenges. The upper limit $T$ for the inner loop was consistently set at 50 across all tasks.

Regarding response generation, earlier tokens in a sequence have a more pronounced influence on the overall arrangement. To address this, our node structure is designed to vary the token counts at different positions in the sequence. The structure follows a pattern of 1, 2, 4, and 10 tokens, with subsequent nodes containing 10 tokens each.

Reviewer comment (Experiments):
In the inner loop, could alternative LLMs be utilized for conducting evaluation? What are your thoughts on the potential benefits of employing multi-agent debating for self-evaluation?

Author response:
Using alternative LLMs for evaluation in the inner loop is an interesting idea. It could reduce biases, especially if the evaluation LLMs differ from the generation model. It is important to note, though, that if the models are similar, the extent of bias reduction might be limited. Additionally, this approach increases memory overhead due to the need to store both generation and evaluation models.

Reviewer comment (Experiments):
RAIN employs the embedding of pre-trained Sentence-BERT for calculating cosine similarity. How do you perceive the possibility of replacing it with the embedding from the LLM itself?

Author response:
Thank you for your comment. This method has the advantage of not requiring an additional model or extra computation during feature extraction. It is noted that the significant size difference between Sentence-BERT and LLMs results in a minimal difference in computational cost. We conducted experiments under the settings of our ablation study for this aspect. In our experiments, we utilized the last token's feature from the last layer's hidden state of the LLM, as it encapsulates the context of the entire sentence, following the standard approach for sentence classification with LLMs. Our experimental results are as follows:

| Embedding           | None | LLMs itself | Sentence-BERT |
|:-------------------:|:----:|:-----------:|:-------------:|
| Attack Success Rate↓| 25%  | 22%         | 19%           |

The results show that while using LLMs for embedding extraction outperforms omitting embedding-related mechanisms (Dynamic Node Addition \& Similarity Update), it is less effective than utilizing Sentence-BERT. This is likely due to Sentence-BERT being specifically trained for sentence embeddings, thereby providing higher-quality embeddings compared to LLM features, which are primarily designed for next-token prediction.

Reviewer comment (Presentation):
While I can see its interest, I believe the robustness experiment should be better introduced and motivated in the paper.

Author response:
The significance of robustness in LLM alignment is highlighted by a study [1] introducing the Greedy Coordinate Gradient (GCG). This study reveals that even well-aligned models, including ChatGPT, are vulnerable to harmful responses under GCG attacks, underscoring the importance of ensuring safety against adversarial attacks.

Reviewer comment (Experiments):
Similarly, truthfulness can obviously be linked to harmlessness — still, I would have liked this to be discussed to better motivate the experiment.

Author response:
LLMs sometimes generate responses that are not truthful, which can potentially mislead users and lead to serious consequences. In fields like healthcare, accuracy in factual information is extremely sensitive. When applying LLMs to these areas, special attention must be paid to the truthfulness of their outputs. Truthfulness is also a current focus in LLM alignment research, with studies such as LLaMA2-chat [2] and InstructGPT [3] both investigating and testing for truthfulness.

Reviewer comment (Evaluation):
While I understand that this may already be common practice, having almost all of the production of your model evaluated by pre-existing models — which, furthermore, are not even public (nor is their training data) — is something I believe to be an issue and should be at least discussed in the paper.

Author response:
To address this potential limitation, we have not only utilized evaluations from pre-existing models but have also conducted human evaluations to validate our findings. The results of these human evaluations are presented in Table 8 on page 9 of our original manuscript. These results demonstrate a close alignment between evaluations conducted by GPT-4 and those by human assessors, indicating consistency in the assessment quality. Additionally, recent research [1] supports the reliability of GPT-4 evaluations, further validating our approach. In our experiments conducted on AdvBench [2], we used the same metrics as those in the original paper that introduced this benchmark. These metrics are not determined by model judgment but are based on explicit, predefined rules.

Reviewer comment (Experiments):
The method completely relies on a self-evaluation measure, in the form of a prompt. This dimension of the method, which to me seems like is the most capital, stays completely uninvestigated. Have other prompts been tested? With a different keyword (harmful/harmless)? How do naïve baselines (using a pre-trained binary classifier in place of self-evaluation, or using the current prompt without the inner loop but in the context of naive generate and evaluate) compare in terms of running time and performance?

Author response:
Thank you for your suggestion. We acknowledge the importance of this aspect and agree that it could enrich our ablation study. Our original prompt already incorporates both "harmful" and "harmless" as key terms, as demonstrated in the prompt example on page 5 of our paper. To further investigate the sensitivity to variations in prompts, we conducted an additional experiment using the following alternative prompt: "This is a conversation between a human and an AI assistant. The assistant's response should comply with ethical guidelines, comply with the law, and avoid providing information that could cause harm or adverse effects. Do you think the assistant's response is harmful?" The results of this experiment are as follows:

| Prompt                  | Original | Modified |
|-------------------------|----------|----------|
| Attack Success Rate↓    | 19%      | 18%      |

Our experimental findings indicate that the utilization of different prompts with similar thematic content does not significantly alter the results, suggesting that the system’s performance is robust to prompt variations.

Reviewer comment (Presentation):
I believe Figure 2 should be capital to the understanding of your method — however, the text helped me in understanding the figure. I believe you should try to simplify what you display for the first contact of the reader with your method.

Author response:
Thank you for your valuable feedback. We provided a simplified diagram, which abstracts away the internal details and utilizes a concrete example for clarity, in the “Supplementary Material” section of our OpenReview submission, within the .zip file, under the file name "Simplified\_Diagram.pdf". The caption for this diagram is as follows: "A simplified explanation of RAIN. RAIN switches between a forward generation phase and a backward rewinding phase, incorporating a self-evaluation stage in between to accomplish self-alignment. The method mirrors human behavioral patterns: contemplating, weighing, and reflecting on the consequences before speaking. Notably, the method operates without the need for extra data and abstains from model updates."

Reviewer comment (Presentation):
A large part of your citations are preprints — if those papers have been published, please update their reference.

Author response:
Thank you for your valuable suggestion. We have thoroughly reviewed all our references and identified four papers that have since been published. These publications are:

- Cho, Woon Sang, et al. "Towards Coherent and Cohesive Long-form Text Generation." Proceedings of the First Workshop on Narrative Understanding. 2019.

- Kreutzer, Julia, et al. "Can Neural Machine Translation be Improved with User Feedback?." Proceedings of NAACL-HLT. 2018.

- Lin, Stephanie, Jacob Hilton, and Owain Evans. "TruthfulQA: Measuring How Models Mimic Human Falsehoods." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

- Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We are committed to updating these citations in the forthcoming version of our manuscript.

Reviewer comment (Experiments):
While I appreciate your ablation studies, I believe those aspects of your method are all well-motivated. If you did not implement ablation studies for other mechanisms in your method, did you have any preliminary results showing the importance of them, or motivating your choices?

Author response:
Indeed, other mechanisms are intricately interconnected, making them infeasible to isolate for individual ablation studies. For instance, the "forward step" relies on the attributes updated in the "evaluation and attribute update step." Removing or altering the "evaluation and attribute update step" would render the "forward step" completely ineffective, as it directly relies on the updates from the previous step. The motivation behind other mechanisms in RAIN is to enhance search efficiency by heuristically exploring areas more likely to lead to optimal generation, utilizing scores from simulated generations.