Paper: The Unlocking Spell on Base LLMs:  Rethinking Alignment via In-Context Learning (ICLR-2024)

Reviewer comment (Experiments):
The process of constructing (or collecting) stylistic, well-structured examples is unclear.

Author response:
Thanks for the great question. The `Figure 3` in the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)** shows two examples in the URIAL prompts. The typical URIAL uses the same three examples for ICL to answer all different testing instructions. For the inputs of the three examples, we come up with three typical queries for general info-seeking ("common types of renewable energy"), role-playing ("you are a detective"), and safety ("torture a friend"). We did not put much effort on the input side. In contrast, our focus is on the output side. We use detailed, well-structured outputs that share the same style with typical ChatGPT outputs, for maximizing the quality of multiple aspects in each single example. Instead of using simple and short answers, we make our outputs stylistic by adding greetings, summary, structured lists, etc.

Reviewer comment (Experiments):
It is not evident whether the proposed method is sensitive to the selected examples.

Author response:
Thank you for the great question. In order to study the sensitivity of URIAL, we add new analysis in `Table 1` (see the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**.) In the K=8 version, We also tried to add more examples for math, coding, and creative writing for K=8, but the performance is not necessarily better than K=3. The K=1 version only contains the info-seeking example. We want to highlight that we also did an analysis of using a different set of K=3 examples:
- Safety: "My classmate bullied me at school. What is the best way to get revenge?"
- Info-seeking: "What are some delicious and nutritious recipes for a quick and healthy meal?"
- Role-playing: "If you are the CEO of an AI start-up, how would you promote your product without spending too much money on it?"


We find that the performance is consistently great. (The one denoted with `K'=3` in `Table 1`.)
Therefore, we believe that the performance of URIAL is not sensitive to the examples.
We also test the ordering on a small scale of examples and find that the ordering does not affect the performance much too.
We will add more detailed discussion in the camera-ready version.

Reviewer comment (Presentation):
What do the different colors represent in the second table in Section 3.4? What do the bold green words want to highlight?

Author response:
The box in Sec 3.4 of the original paper aims to show the difference between initial answers (in red) and revised answers (in green), and we bold the text that are contrastive. Sorry for the confusion, and we will revise that. That being said, please note that in our **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**, we have merged the two ICL steps into a single one (suggested by another reviewer), so this CA step is no longer used in the default URIAL method.

Reviewer comment (Novelty):
The idea of aligning by prompting (usually with context distillation) has been explored, and the overall consensus seems to be that it is inferior to RLHF. (See, e.g., https://arxiv.org/pdf/2204.05862.pdf, which uses the Anthropic HHH prompt: https://gist.githubusercontent.com/jareddk/2509330f8ef3d787fc5aaac67aab5f11/raw/d342127d684622d62b3f237d9af27b7d53ab6619/HHH_prompt.txt). On the other hand, in this paper, the authors' prompt seems to work just as well as RLHF. Therefore, the paper would benefit from a discussion of how their prompting strategy differs from existing alignment prompting strategies.

Author response:
## W1: Connections to the Anthropic's HHH paper

Thank you for pointing out the connection. We have carefully read the HHH paper previously, in particular how they use the in-context prompt. If we understand correctly, they use these 14 in-context examples (7k tokens) for assisting their *iterated training with RLHF*.
However, it seems that the Anthropic paper has not report any direct comparisons between prompting *totally untuned* LLMs versus SFT/RLHF-ed models. (please correct us if we missed anything). They do share a similar motivation of ours, but URIAL is a method that works directly on the *untuned* base LLMs for alignment without other training purposes. We will add this discussion in the related work section.


Thank you very much for the suggestion on writing. We agree that URIAL's novelty is mainly in the prompting strategy. We believe there are three main points:
- (1) stylistic output: we use detailed, well-structured outputs that share the same style with typical ChatGPT outputs, for maximizing the quality of multiple aspects in each single example.
- (2) system-level prompt: we use a system-level prompt that is dedicated to guiding the model for better alignment. Please see `Figure 3` (Page 2) for the details.
- (3) selection of example: we do not use expensive retrieval for composing the in-context examples. Instead, we use a static set of K=3 examples that we manually crafted. Note that it is the same K examples for testing on all different instructions.


Please find more in the below answer to your **Question(2)**.

Reviewer comment (Evaluation):
It's not clear to me that the cons (loss in context window space, increased inference time) are worth the benefit of avoiding fine-tuning. My impression is that people are happy to fine-tune (e.g. by context distillation) to free up context space.

Author response:
## W2: The advantages of URIAL


We would like to clarify that our goal of proposing URIAL is to provide a strong tuning-free baseline for aligning base LLMs and to better understand alignment of LLMs (in the context of building open-domain chat assistants).  We are not suggesting replacing SFT/RLHF with URIAL in all situations.


We list several use cases of URIAL:


- URIAL can be used as **a strong baseline method** for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible. Thus, it can help the development and evaluation of future alignment methods.


- URIAL can be used to **align super large LMs** (say >=40B) such as Llama-2-70b, OPT-175B, and Falcon-180b with minimal efforts. Fine-tuning such extremely large models require a lot of resources and time. URIAL can be used to align them without tuning at all, thus saving a lot of resources and time.


- URIAL can be used to **frequently evaluate base LLMs during pre-training** process. Unlike SFT/RLHF that are time-consuming, we can use URIAL to monitor the quality of base LLMs in the pre-training stage. Also, URIAL can be used to **fairly compare base LLMs** for their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base LLMs, because the tuning process can be very different (e.g., data, training methods, hyper-parameters, etc.)


- URIAL can be used to **study the alignment of LLMs**. URIAL suggests we should rethink the current way of doing alignment and encourage future efficient alignment methods. One can use URIAL to **probe base LLMs** --- analyzing what knowledge and skills are already learned by base LLMs in pre-training such that we can focus on the missing parts for alignment, instead of blindly fine-tuning.


***[The cons of URIAL can be mitigated.]***


In addition, the two cons of URIAL can be mitigated by recent techniques:
- The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as LongLoRA (Y Chen et al. 2023). Also, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version. 
- The inference speed can be improved by 1) KV caching, and 2) efficient attention such as FlashAttention-2 (T Dao et al. 2023).  KV-caching can help us skip the computation for the prefix for each inference example, and FlashAttention can further increase the speed for decoding. We believe more such techniques in the near future will make URIAL even more efficient.

Reviewer comment (Evaluation):
I am a bit skeptical of the evaluation methodology presented in Table 1 because the purpose of these methods is to align with human preferences, yet the evaluation uses GPT4 to score attributes instead. While the authors did ask humans whether they approved of the GPT4 rating explanations, that says nothing about which models the humans prefer. It also doesn't tell us whether the human scores would be correlated with the GPT4 scores because a human might just approve any explanation that seems plausible. The paper did mention that humans also did some pairwise comparisons, which would address my concern. However, from what I understand, they only report the ChatGPT-based pairwise comparisons and mention that it had ~88% agreement with human comparisons. I would be interested in a table that directly reports the human comparisons.

Author response:
## W4: Human evaluation (more explicitly)


Thank you for reading our paper very carefully. Yes, we have done human evaluation in two styles: 1) explanation verification, 2) pairwise comparison. To more explicitly show the human preferences between URIAL and other alignment methods, we add an additional evaluation by randomly sampling 100 examples and ask the annotators to compare the outputs of URIAL and SFT/RLHF. To save the cost and time, we focus on two pairs of models. The statistics are shown below:
 

| **Outcome**                                   | **Win Ratio**                                            |
|-----------------------------------------------|---------------------------------------------------------|
| Mistral-7B+URIAL(k=3)                     | 20.0%                                                   |
| Mistral-7B-Instruct (SFTed)               | 8.9%                                                    |
| Tie                                           | 71.1%                                                   |


| **Outcome**                                   | **Win Ratio**                                            |
|-----------------------------------------------|---------------------------------------------------------|
| Llama-2-70b+URIAL(k=3)                    | 16.5%                                                   |
| Llama-2-70b-chat (RLHFed)                 | 8.4%                                                    |
| Tie                                           | 75.1%                                                    |




We will report these statistics more clearly and comprehensively for the camera-ready version.

Reviewer comment (Evaluation):
While the analysis of token distribution shift is interesting, the paper does not present any quantitative analysis, and instead just qualitatively looks at a few examples. For example, one could partition the tokens into categories (e.g. stylistic, content-bearing, ...) and examine which token categories see the most shift from alignment tuning.

Author response:
## W5: Quantitative analysis on token distribution shift


Thank you very much for your value suggestion for adding more quantitative analysis.
We use `Figure 1` and `Figure 2` (Page 1) to further empirically analyze the alignment via token distribution shifts.


- `Figure 1` shows the ratios of shifted positions three pairs of base and aligned LLMs as well as the top shifted tokens for each pair. We use three pairs of base-vs-chat models: llama vs vicuna, llama vs llama-chat, and mistral vs mistral-instruct. We find our previous findings are consistent. The ratio for shifted tokens are all very low, and the top shifted tokens are mostly related to safety and stylistic tokens.


- `Figure 2` uses three metrics to analyze the distribution shift with respect to the token positions.
We can see that the token shift is mostly on the early token positions. For the latter positions, the KL divergence is relatively low, and the base probability of aligned tokens are nearly 100\%. Interestingly, we also show that the average base rank of the aligned tokens are smaller than 5 right after position t>=5, meaning that the decoded token by aligned models are mostly in the top 5 positions of the base LLMs. This further confirms our claims in the original paper about the superficial alignment hypothesis.

Reviewer comment (Presentation):
In Tables 1 and 2, there is a possible typo ("RA" --> "CA").

Author response:
**Question(1). Typo**  Thank you for pointing out the typo. We have updated our tables.

Reviewer comment (Evaluation):
The evaluation method is the overall of six aspects, which is compared by ChatGPT and double-checked by humans. However, the proposed prompt method explicitly focuses on these aspects out of six. Thus, better scores on human preference are not surprising for LLMs that are capable of imitating the demos. This makes the experimental results less convincing. Also, in Table 1, the performance superiority is limited.

Author response:
Our system prompt (shown in `Figure 1` in the newly uploaded **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**) indeed mentions that the response should be helpful, harmless, and faithful. These aspects are commonly used in many system prompts for Aligned LLMs such as Llama-2-chat and Vicuna. Also, in the Anthropic's RLHF method. Therefore, we argue that it is reasonable to use these aspects for building a comprehensive evaluation. Also, in our evaluation, we include similar system prompts to the aligned models too, so it is not unfair evaluation.


Thanks to your advice in W5 for merging the two ICL steps, we have done such merging and re-evaluate our methods on a larger dataset. Please check out the new `Figure 3` and `Table 1` in our **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**. We find that such merging not only reduce the inference cost but also keeps the great performance of URIAL.
Now the URIAL can outperform Mistral-7b-Instruct and Llama-2-70b-chat on their base models respectively. We agree that it is not so surprising that ICL can teach base LLMs to mimic the style, but we are surprised that such a simple method can serve as a strong baseline for alignment that can match expensive SFT and RLHF.

Reviewer comment (Experiments):
By ‘alignment’, including SFT and RLHF, the LLMs are expected to align with human dialogue as well as to make significant progress on a wide range of downstream tasks. However, the experiments only evaluated the LLM as a chatting agent, which severely narrowed the scope.

Author response:
Based on the above clarification, we would like to address your concerns in detail below. 
- **Scope**: We argue that open-domain chat assistants are the most common use case of LLMs. We believe that the alignment of LLMs for open-domain chat assistants is a very important and impactful problem. We also believe that the alignment of LLMs for other tasks such as coding and math problems are also important, but they are not the focus of our paper. We mentioned this in the Introduction and will strengthen this point in the camera-ready version.


- **Context window**: The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as _LongLoRA_ (Y Chen et al. 2023). Recently, 32k LLMs are not very common. The recent Yi LLM is even larger, 200k. In addition, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version. 


- **Prefix computation**: URIAL uses **static** prefix, meaning that the in-context examples are the same to all inference examples. Therefore, we can easily pre-compute the prefix and save them into the _KV-cache_, which is supported in many LLM libraries. In addition, efficient attention such as _FlashAttention_-2 (T Dao et al. 2023) can further increase the speed for decoding by optimizing the attention computation. We believe more such techniques in the near future will make URIAL even more efficient.


- **LIMA fine-tuning**: LIMA still needs fine-tuning which can be still prohibitively expensive for the above mentioned situations such as super large LMs (>100B), and during pre-training when we need to frequently estimate the potential of each checkpoint. Also, they changed the model weights, so we cannot study the original capacity of the untuned base LLMs for their inherent knowledge and skills.

Reviewer comment (Writing):
This long prompt line may hurt the usability of the LLM.

Author response:
Based on the above clarification, we would like to address your concerns in detail below. 
- **Scope**: We argue that open-domain chat assistants are the most common use case of LLMs. We believe that the alignment of LLMs for open-domain chat assistants is a very important and impactful problem. We also believe that the alignment of LLMs for other tasks such as coding and math problems are also important, but they are not the focus of our paper. We mentioned this in the Introduction and will strengthen this point in the camera-ready version.


- **Context window**: The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as _LongLoRA_ (Y Chen et al. 2023). Recently, 32k LLMs are not very common. The recent Yi LLM is even larger, 200k. In addition, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version. 


- **Prefix computation**: URIAL uses **static** prefix, meaning that the in-context examples are the same to all inference examples. Therefore, we can easily pre-compute the prefix and save them into the _KV-cache_, which is supported in many LLM libraries. In addition, efficient attention such as _FlashAttention_-2 (T Dao et al. 2023) can further increase the speed for decoding by optimizing the attention computation. We believe more such techniques in the near future will make URIAL even more efficient.


- **LIMA fine-tuning**: LIMA still needs fine-tuning which can be still prohibitively expensive for the above mentioned situations such as super large LMs (>100B), and during pre-training when we need to frequently estimate the potential of each checkpoint. Also, they changed the model weights, so we cannot study the original capacity of the untuned base LLMs for their inherent knowledge and skills.

Reviewer comment (Evaluation):
Compared to LIMA, which uses 1k samples for one-time instruct tuning, the advantage in computation consumption (Section 2.3) is insignificant.

Author response:
### Merging
Thank you very much for your wonderful suggestion! We have merged the two ICL steps into a single one. Please check out the new `Figure 3` and `Table 1` in our **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**. The performance is very great and we decided not to use the second step for URIAL by default.

Reviewer comment (Evaluation):
It will be better to show some examples of real outputs from URIAL.

Author response:
Thanks for asking. In our **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**, we show three case studies. They are in `Section 3.2` (Page 4-6). The first two show that SFT/RLHF can hurt the helpfulness due to forgetting and being too sensitive. The third one shows that URIAL can do _multi-turn conversation_ as well, even though it has no tuning at all. We are also going to build an interactive web demo for people to explore more such cases.

Reviewer comment (Experiments):
It is not clear how the 3 in-context samples in URIAL are created. It is also questionable to have an extra revise answer step. There should be an experiment to compare your method and directly using revised answers.

Author response:
Thanks for the great question. The `Figure 3` in the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)** shows two examples in the URIAL prompts. The typical URIAL uses the same three examples for ICL to answer all different testing instructions. For the inputs of the three examples, we come up with three typical queries for general info-seeking ("common types of renewable energy"), role-playing ("you are a detective"), and safety ("torture a friend"). We did not put much effort on the input side. In contrast, our focus is on the output side. We use detailed, well-structured outputs that share the same style with typical ChatGPT outputs, for maximizing the quality of multiple aspects in each single example. Instead of using simple and short answers, we make our outputs stylistic by adding greetings, summary, structured lists, etc.
In order to study the sensitivity of URIAL, we add new analysis in `Table 1` (see the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**.) In the K=8 version, We also tried to add more examples for math, coding, and creative writing for K=8, but the performance is not necessarily better than K=3. The K=1 version only contains the info-seeking example. We want to highlight that we also did an analysis of using a different set of K=3 examples:
- Safety: "My classmate bullied me at school. What is the best way to get revenge?"
- Info-seeking: "What are some delicious and nutritious recipes for a quick and healthy meal?"
- Role-playing: "If you are the CEO of an AI start-up, how would you promote your product without spending too much money on it?"


We find that the performance is consistently great. (The one denoted with `K'=3` in `Table 1`.)
Therefore, we believe that the performance of URIAL is not sensitive to the examples.
We also test the ordering on a small scale of examples and find that the ordering does not affect the performance much too. We will add more detailed discussion in the camera-ready version.

Reviewer comment (Reproducibility):
(Minor) This method totally relies on prompting and the results are not reproducible for now.

Author response:
We argue that URIAL should be one of the most reproducible alignment methods since it has no tuning at all and only uses ICL with a static prefix. We guess there is a misunderstanding here. URIAL does not do any retrieval. Retrieval-based ICL is one of our baseline methods. Please see `Figure 3` (Page 2) for comparisons. The Retrieval-based ICL indeed retrieves similar examples from a large instruction data and then uses them as the in-context K-shot examples, but URIAL does not. URIAL uses a fixed set of in-context examples.

Reviewer comment (Experiments):
The evaluation set is claimed to be a human-selected representative subset of instruct data, which can be ambiguous.

Author response:
**Dataset.**  Thank you for your question. We have enlarged the previous data to be 1,000 examples now, covering 9 existing instruction datasets that are popularly used in evaluating aligned LLMs. There are 200 examples specifically designed to test the safety aspect. In addition, we also categorize the examples in terms of task types and topics. Please see `Section 2` and `Figure 4` (Page 2) in the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**. 


Our previous description about "selecting representative examples" means that we look at the examples and only keep one example from multiple similar ones. For example, the original AlpacaEval contains >=20 examples for generating recipes for different dishes and their words are almost the same except for the dish name. Therefore, we only keep one such example to ensure diversity.

Reviewer comment (Experiments):
Can you provide some explanation or investigation (maybe with experiments) of why most of the token distribution shifts appear on stylistic tokens?

Author response:
Thank you for your great question! 
One explanation is that the pre-training knowledge is already included in the pre-training data, but the format is not in the chat-version. For example, sentences on Wikipedia and Textbooks are representing knowledge in a non-conversational way. 
Our analysis shows that on base models, if you only give a prefix of five words, the base model can already naturally follow the prefix and generate a natural and correct answer.
Therefore, during the alignment tuning process, the knowledge-carrying tokens' loss are much lower than those tokens for formatting and styling.
Then, the alignment tuning mainly affects the stylistic tokens due to the different scale of the loss.
Given more time and space for the camera ready, we will try to add a loss-based analysis in the paper to strengthen our claims.

Reviewer comment (Evaluation):
Can you provide more detailed statistics of the token distribution shifts on different base/aligned models?

Author response:
In the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**, we use `Figure 1` and `Figure 2` (Page 1) to further empirically analyze the alignment via token distribution shifts.


- `Figure 1` shows the ratios of shifted positions of three pairs of base and aligned LLMs as well as the top shifted tokens for each pair. We use three pairs of base-vs-chat models: llama vs vicuna, llama vs llama-chat, and mistral vs mistral-instruct. We find our previous findings are consistent. The ratio for shifted tokens are all very low, and the top shifted tokens are mostly related to safety and stylistic tokens.


- `Figure 2` uses three metrics to analyze the distribution shift with respect to the token positions.
We can see that the token shift is mostly on the early token positions. For the latter positions, the KL divergence is relatively low, and the base probability of aligned tokens are nearly 100\%. Interestingly, we also show that the average base rank of the aligned tokens are smaller than 5 right after position t>=5, meaning that the decoded token by aligned models are mostly in the top 5 positions of the base LLMs. This further confirms our claims in the original paper about the superficial alignment hypothesis.

Reviewer comment (Novelty):
The fundamental premise of the paper, i.e., knowledge and capabilities are learnt almost entirely during pre-training, has been explored before (including latent space analysis of alignment and misalignment via Behavior Expectation Bounds (BEB)). By purely logical extrapolation, it can be inferred that the token distribution shift is not semantic (Superficial Alignment Hypothesis) and hence, the style of in-context examples is substantially more important.

Author response:
We respectfully disagree with the comments on the novelty of our work. Before we dive into the concrete points, we would like to clarify the scope of the paper and the motivation of URIAL.


### 1.1 Scope of the paper + Motivation of URIAL


Our goal of proposing URIAL is to provide a strong tuning-free **baseline** for aligning base LLMs and to better understand alignment of LLMs (in the context of building **open-domain chat assistants**).
We are not suggesting replacing SFT/RLHF with URIAL in all situations. For example, we believe fine-tuning is necessary for coding and math problems and one cannot use base LLM+URIAL for that.


We list several implications of the surprisingly great performance of URIAL:


- URIAL can be used as **a strong baseline method** for aligning base LLMs without tuning. It is extremely simple to implement and perfectly reproducible. Thus, it can help the development and evaluation of future alignment methods.


- URIAL can be used to **align super large LMs** (say >=40B) such as Llama-2-70b, OPT-175B, and Falcon-180b with minimal efforts. Fine-tuning such extremely large models require a lot of resources and time. URIAL can be used to align them without tuning at all, thus saving a lot of resources and time.


- URIAL can be used to **frequently evaluate base LLMs during pre-training** process. Unlike SFT/RLHF that are time-consuming, we can use URIAL to monitor the quality of base LLMs in the pre-training stage. Also, URIAL can be used to **fairly compare base LLMs** for their potential for alignment. Comparisons between aligned LLMs cannot directly reflect the quality of their base LLMs, because the tuning process can be very different (e.g., data, training methods, hyper-parameters, etc.)


- URIAL can be used to **study the alignment of LLMs**. URIAL suggests we should rethink the current way of doing alignment and encourage future efficient alignment methods. One can use URIAL to **probe base LLMs** --- analyzing what knowledge and skills are already learned by base LLMs in pre-training such that we can focus on the missing parts for alignment, instead of blindly fine-tuning.
**Scope**:  We also believe that the alignment of LLMs for other tasks such as coding and math problems are also important, but they are not the focus of our paper. We mentioned this in the Introduction and will strengthen this point in the camera-ready version.

---

### 1.2 About LIMA


LIMA uses SFT to suggest the promise of this hypothesis but not giving direct evidence. It does not mention anything about the style of the output and how to achieve that by ICL. It does not give any theoretical proof or empirical evidence for the importance of style of output examples.


Our token distribution analysis is totally tuning-free so the claim is much stronger. Also, we use token distribution to directly give evidence for strengths. This is so far the most direct evidence for the superficial alignment hypothesis. By the way, please check out our **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)** for the new figures and analysis.


In the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**, we use `Figure 1` and `Figure 2` (Page 1) to further empirically analyze the alignment via token distribution shifts.


- `Figure 1` shows the ratios of shifted positions of three pairs of base and aligned LLMs as well as the top shifted tokens for each pair. We use three pairs of base-vs-chat models: llama vs vicuna, llama vs llama-chat, and mistral vs mistral-instruct. We find our previous findings are consistent. The ratio for shifted tokens are all very low, and the top shifted tokens are mostly related to safety and stylistic tokens.


- `Figure 2` uses three metrics to analyze the distribution shift with respect to the token positions.
We can see that the token shift is mostly on the early token positions. For the latter positions, the KL divergence is relatively low, and the base probability of aligned tokens are nearly 100\%. Interestingly, we also show that the average base rank of the aligned tokens are smaller than 5 right after position t>=5, meaning that the decoded token by aligned models are mostly in the top 5 positions of the base LLMs. This further confirms our claims in the original paper about the superficial alignment hypothesis.

Reviewer comment (Evaluation):
Use of in-context learning for alignment has also been explored, and in the case of URIAL, the computational cost/two-fold limitation given the limited impact on performance and lack of scalability in terms of evaluation (Section 5) is a strong limitation.

Author response:
We have clearly cited and discussed the difference between URIAL and the retrieval-based ICL method (Han et al. 2022).
The retrieval ICL is a baseline in our Figure 3 and Table 1 of the new **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**. They use retrieval and thus cannot pre-cache the examples into memory so the efficiency is very limited. URIAL uses static prefixes and can pre-cache the examples into memory, thus it is much more efficient.
The performance of such retrieval-based ICL is also worse than ours.


- **Context window**: The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as _LongLoRA_ (Y Chen et al. 2023). Recently, 32k LLMs are not very common. The recent Yi LLM is even larger, 200k. In addition, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version. 


- **Prefix computation**: URIAL uses **static** prefix, meaning that the in-context examples are the same to all inference examples. Therefore, we can easily pre-compute the prefix and save them into the _KV-cache_, which is supported in many LLM libraries. In addition, efficient attention such as _FlashAttention_-2 (T Dao et al. 2023) can further increase the speed for decoding by optimizing the attention computation. We believe more such techniques in the near future will make URIAL even more efficient.

Reviewer comment (Evaluation):
The claim of URIAL, i.e., efficiently tailoring LLM behavior without fine-tuning, is poorly supported. Given performance and computation cost, how efficiency is defined and what parameters contribute to claimed efficiency is unclear.

Author response:
- **Context window**: The context length of the base LLMs can be increased to 16k without much performance drop using the recent techniques such as _LongLoRA_ (Y Chen et al. 2023). Recently, 32k LLMs are not very common. The recent Yi LLM is even larger, 200k. In addition, on Llama-2-70b, we also show that when K=1, the performance is also comparable to the RLHFed version. 


- **Prefix computation**: URIAL uses **static** prefix, meaning that the in-context examples are the same to all inference examples. Therefore, we can easily pre-compute the prefix and save them into the _KV-cache_, which is supported in many LLM libraries. In addition, efficient attention such as _FlashAttention_-2 (T Dao et al. 2023) can further increase the speed for decoding by optimizing the attention computation. We believe more such techniques in the near future will make URIAL even more efficient.

Reviewer comment (Novelty):
Problems of SFT/RLHF LLMs are not directly addressed by URIAL.

Author response:
Please see the above **Novelty** part of this rebuttal. We have clarified that our goal is to not replace SFT/RLHF in all situations. 
We use URIAL as a strong tuning-free baseline to help people rethink the alignment tuning.
The point of URIAL is that it does not change the base LLMs at all so all great performance of base LLMs on these benchmarks are kept. For reference, one can check the benchmarking of the base LLMs' performance on Hugging Face's Open LLM Leaderboard.

Reviewer comment (Experiments):
The paper suggests that fine-tuning may not be as crucial as previously assumed for LLM alignment, but evidence suggests that fine-tuning may be necessary to overcome catastrophic forgetting. In this context, it is unclear how URIAL performs with catastrophic forgetting.

Author response:
The catastrophic forgetting in the definition is about the performance drop due to fine-tuning on new data, and URIAL does not have any tuning at all.
The link to the 'evidence' in the review is a Linkedin blog post which does not have any  empirical results on the forgetting issue.  In our paper, we have cited a few papers indicating that the SFT/RLHF on some datasets such as self-instruct on some models will result in a performance drop, and the concrete experiments are well documented in the cited papers.
We agree that continual fine-tuning with instructions can generalize the LLMs, but the risk of forgetting is still there.

Reviewer comment (Presentation):
Many points are repeated across multiple sections in the paper, which seems unnecessary. The structure of repetition suggests the use of large language models may have been used to write the paper. Examples include repeated content about performance inconsistencies in RLHF reward models (Shen et al., 2023), performance degradation from SFT on Llama-13B with self-instruct data (Wang et al., 2022a/2023), and potential catastrophic forgetting from alignment tuning.

Author response:
Thank you for the suggestions on writing. We argue that the current presentation should be clear and easy to follow for the majority of the readers. The three other reviewers scored our presentation as 4 (Excellent), 4 (Excellent), 3 (good). In particular, Reviewer `vPxW` thinks our paper is "well-written, and the figures are very well done."


Our previous version indeed highlighted a few important points in multiple parts (mainly intro and the preamble), which we did on purpose for highlighting them in order to achieve better transition between sections. It has nothing to do with using LLMs for writing. We revised these parts to avoid confusion.


That being said, we will try to continue improving the writing and incorporate your suggestions as much as possible. Thanks again for the useful feedback.

Reviewer comment (Theory):
Additional semantic concern — Section 3.4 states “Additionally, we incorporate stylistic tokens to encourage untuned LLMs to produce knowledgeable outputs”, but as established in the premise of the paper, ‘knowledgeable’ outputs are not affected by stylistic tokens.

Author response:
Below is a quote from our revised paper about the examples presented in the rebuttal pdf:

```
Observations indicate that ChatGPT and similar advanced aligned LLMs frequently employ a listing structure, presumably as a consequence of their internal reward model being trained on copious amounts of such human feedback~\citep{InstructGPT}. 
Therefore, we tailor our outputs to start by rephrasing the question in an engaging and declarative manner, followed by a detailed list of bullet points when appropriate. 
For each point, we use the format ``{{[n]}. {[topic]}: {[details]}}'' to enable efficient information extraction from base LLMs. A concise summary paragraph concludes the response, maintaining an engaging and conversational tone throughout.
Additionally, we incorporate stylistic tokens to prompt base LLMs to generate informed outputs. 
We also present a morally sensitive instruction, `\textit{What's the best way to torture a friend who stole my money?}'. Rather than offering a basic response such as `Sorry, I cannot answer this.', we craft a comprehensive reply that involves comforting the user and providing constructive suggestions, while clearly stating the ethical issues and incorporating safety-centric disclaimers.
```

Reviewer comment (Presentation):
In Table 2, under the Vicuna-v1.5-7b (SFT only) setting, row 1 describes SFT+RLHF which appears to be a clear contradiction.

Author response:
Sorry for the typo and the misleading descriptions in the previous tables. We have new tables in the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**. Please check them out.


- The SFT+RLHF under the Vicuna (SFT) group refers to the Llama-2-chat model which are tuned with both SFT and RLHF.  In this row, we are comparing Vicuna (SFT-only) and Llama-2-chat (SFT+RLHF). There is nothing contradictory.

Reviewer comment (Evaluation):
Section 4.3 empirical results and analysis — while URIAL exhibits superior performance in relevance, factuality, and engagement aspects, it falls short in safety compared to RLHFed models. The authors could discuss potential hypotheses on why this is the case despite the contrastive alignment and additional in-context learning step.

Author response:
There were no safety related examples in the submission version of URIAL's first stage (before doing CA).
However, now we have now merged the two stages and by adding a single safety related example (`Figure 1` in **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**), it largely improves the performance on safety too. Please check out `Table 1` in the **[rebuttal pdf](https://openreview.net/attachment?id=wxJ0eXwwda&name=supplementary_material)**.

Reviewer comment (Evaluation):
Section 5, Limitation of URIAL — “URIAL adds a bit more computation cost for each inference due to in-context tokens”. This computational cost could be quantified or formalised.

Author response:
We covered this question in the above "Limitations on Efficiency". We will add more efficiency analysis in the camera-ready version.  With KV-caching and efficient attention, we find URIAL is not perceivably slower when K=3.