Paper: Aligner: One Global Token is Worth Millions of Parameters When Aligning LLMs (ICLR-2024)

Reviewer comment (Evaluation):
The interpretation of the win ratio for Aligner 1 (1 token) in Table 1 is given as “comparable” with the baselines, but it is actually <0.5, so technically worse than the baselines. This interpretation seems rather too optimistic, it is missing an analysis and a focus on the loss cases, because in practice no one wants to deploy a technique that worsens previous results, perhaps even if there’s a win in parameter efficiency. The trade-off between quality and number of parameters for the new and the standard adapter would have been helpful to compare directly in a plot.

Author response:
We toned down our claim, and we also provided samples of failure cases in an appendix. 

"This isn't always the case. For industrial scenarios where one needs to provide a customized model to every different user (such as assuming a unique character or replying in a customized format), the parameter efficiency is very important when scaling up to tens of thousands or even millions of customized adapters. For Aligner, even with one million adapters, that's just 5 billion extra parameters (around 10G memory), which can still fit into a single 24GB memory GPU along with a 7B model (13G memory) using standard bfloat16 precision without quantization. By contrast, with LoRA or LLaMA-Adapters, this will consume terabytes with millions of adapters and it'll be necessary to offload to secondary storage, which would drastically slow down the inference time. So, our method could be very useful in saving costs. However, thank you for raising this concern as we must expand on this point in our paper. We agree that our "paper should've allocated more space for an investigation of the implications of this small change"."

Reviewer comment (Experiments):
Large portions of the paper are dedicated to reviewing the background, the basics of Llama Adapter and RLHF. Considering that the proposed method is a minor modification, more space should have been rather invested for more exhaustive ablations and deeper analysis of why the method works, how it differs qualitatively from Llama Adapter, and how the hypothesis of separation of form and knowledge can be empirically verified.

Author response:
We believe that we've already largely verified it: The very structure of Aligner -- a minimally sized and globally orthogonal component, along with the fact that its efficacy is on par with other methods, confirms that form functions orthogonally to reasoning/knowledge in LLMs.

However, we do agree that we can further validate our hypothesis. We can also provide a negative example; i.e., for reasoning tasks, Aligner should not exhibit a similar advantage over other higher-parameter and non-global component tuning methods. Since math serves as the most representative reasoning task, we trained them on a math dataset (MetaMath) and tested their performance on a standard benchmark, GSM8K. Aligner indeed does not show a parameter advantage over other methods, thereby validating the hypothesis. 

Please see Item 1 in our above common comment and Section 4.3 of our revised paper.

Reviewer comment (Evaluation):
Why does the 7B model perform better than the 13B model in Table 1? Any hypotheses?

Author response:
This may be due to the variation of GPT 4 evaluation and/or due to the training dynamics of the model as a result of hyper-parameter choices. We didn't expend substantial effort to find optimal hyper-parameters, but they definitely matter to a certain extent, which may account for the slightly better performance.

Reviewer comment (Reproducibility):
Is the code open-sourced or published with the paper?

Author response:
Our code will be open sourced after we tidy it up, but we can release a version quicker if this is deemed necessary.

Reviewer comment (Theory):
Strictly speaking, LLaMA-Adapter (and likewise Aligner) do not prepend prefix tokens. They employ a side-channel of attention over a separate set of tokens. The softmax is compute separately from the actual input tokens (see: Eqn 8), and position encodings are not applied to the LLaMA-Adapter/Aligner keys.

Author response:
You are right that the LLaMA-Adapter and Aligner is not strictly a prefix token method. We have changed the wording to be more rigorous in the Related Work section.

Reviewer comment (Experiments):
The experiment demonstrating the efficacy of single-head Aligner is a simply a handful of examples of generated outputs. This does not meet the bar of rigorous evaluation for a conference paper.

Author response:
3. Regarding the single-head Aligner experiments:

Thank you for your criticism of this section. We have deleted the section for the reasons explained in Item 3 in our above common comment.

Reviewer comment (Experiments):
How does this method perform on standard benchmark tasks?

Author response:
Thank you for your valuable feedback.

We have added some standard benchmark results to address your questions and concerns. However, we would like to offer the following clarifications:

1. The rationale for using GPT as the judge: 

First, this was the default evaluation method in the papers introducing both the Alpaca and Beaver datasets on which we trained Aligner. It was therefore natural to adopt their methods.

Second, to our knowledge, standard datasets like MMLU (general knowledge) or GSM8K (math) have precise correct answers and only care if the final answer is correct. They generally test the *reasoning ability* of a model rather than the *form* of its output.  By contrast, for alignment tasks like Instruction Following and Value Alignment, which are the focus of our paper, the model outputs are open-ended and there is no single right answer. The "form" of the answer is of importance. So, in our case, the gold standard is human judgment. This is also why the default benchmark methods provided by Alpaca and Beaver are both model-based evaluations, since, despite its shortcomings, thus far a powerful model is the closest surrogate to human judgment for such tasks.

However, we agree that it would help in evaluating Aligner more completely if we include some standard benchmarks, but the purpose this can serve would be different. Rather than directly evaluating the "form" alignment task, it helps to see how our Aligner method affects the reasoning ability of the model in comparison to other adaptation methods, and to further test the hypothesis that form functions orthogonally from reasoning or knowledge. This experiment may also help better validate our hypothesis that form functions orthogonally to reasoning/knowledge in LLMs.

2. The standard benchmark results:

First, we added the MMLU evaluation that tests general knowledge without training over its dataset, but using the model finetuned on the Alpaca dataset only, along with the raw LLaMA2 base model's performance. The Alpaca dataset does help reasoning ability to some extent since it contains answers that have step by step reasoning and even some math and coding questions. However, it's primarily aiming to tune the model to follow instructions. Therefore, if form functions orthogonally to reasoning, Aligner should perform worse but there should not be a big gap. The results validate our hypothesis.

The evaluation results:

| Alpaca Finetuned Only |  | MMLU Accuracy |  |
| --- | --- | --- | --- |
| 7B | Raw Base Model | 3588/14042 | 0.2555 |
|  | LLaMA-Adapter | 5037/14042  | 0.3587 |
|  | LoRA | 4958/14042  | 0.3531 |
|  | Aligner 1 | 4025/14042  | 0.2866 |
|  | Aligner 10 | 4613/14042  | 0.3285 |
| 13B | Raw Base Model | 4112/14042 | 0.2928 |
|  | LLaMA-Adapter | 5056/14042 | 0.3601 |
|  | LoRA | 5268/14042 | 0.3751 |
|  | Aligner 1 | 4607/14042  | 0.3281 |
|  | Aligner 10 | 4835/14042  | 0.3443  |

Second, more importantly, we trained different methods over the MetaMath dataset (a math dataset) on the 7B model and evaluated on GSM8K. This was to assess how much reasoning improvement different methods can attain along with their different parameter sizes. If form functions orthogonally to reasoning in LLMs, Aligner should not have an advantage over other methods, unlike in form alignment tasks (Instruction following and Value alignment).  The results validate our hypothesis. Aligner only catches up the performance of LLaMA-Adapter when they have a similar number of parameters. 

| Method  | GMS8K |
| --- | --- |
| Aligner 1 (Param 5k)| 70/1319 0.053 |
| Aligner 10  (Param 42k)| 136/1319 0.1031 |
| Aligner 100  (Param 410k)| 215/1319 0.1630 |
| Aligner 300 (Param 1.2M)| 346/1319 0.2623 |
| LLaMA-Adapter (Param 1.2M)| 334/1319 0.2532 |
| LoRA (Param 4.2M)| 469/1319 0.3556 |
  
If the above rationale and plan are unsatisfactory and you have something else in mind, please let us know what specific evaluation(s) you would like us to perform.

Reviewer comment (Evaluation):
There exist no analysis on the learned parameters especially for tokens.

Author response:
Thank you for your valuable feedback.

We once plotted the distribution of the parameter values of the token, but it is essentially Gaussian and provides no useful insights.

Unfortunately, other common methods like linear classifier probing \cite{linearProbe} are also inapplicable: it requires not only numerous datapoints but also labels for them in order to train this classifier. In our case, we obtain only 1 set of tokens from training over the entire dataset. 

The only meaningful approach seems to be to compare the Aligner token with the LLaMA-Adapter tokens using t-SNE or directly. This could potentially reveal how our globally-shared token relates to the local tokens. We have conducted several comparisons and added the results to the main paper and the supplementary appendix. Here's a list of findings:
1. The Aligner 1 Token embedding is not necessarily the "average" of Aligner 10 Token or the LLaMA-Adapter, which has 300 tokens.
2. When we train the model with a different dataset, the embeddings of Aligner and those of the LLaMA-Adapter remain very close to each other. In fact, approximately half of the numbers in the embeddings are *exactly the same*, and the majority of the rest are still very close. This shows that very little change is needed to dramatically adapt the behaviors of an LLM.
3. The standard deviation of the gating factors in a layer is increasing as the layer goes to top, for both Aligner and LLaMA-Adapter. This may be in align with our general intuition that top layers generally requires larger adaptations. 

The figures are shown in section 4.4, figure 6 in the main paper and Appendix D in the supplement. We also put some analysis that doesn't yield any finding in Appendix D that you can take a look at. We would be pleased to perform any other suitable data analysis that you can recommend to us.

Reviewer comment (Evaluation):
It would be better to quantify the gains by investigating what was learned in the small number of parameters.

Author response:
Thank you for your valuable feedback.

We once plotted the distribution of the parameter values of the token, but it is essentially Gaussian and provides no useful insights.

Unfortunately, other common methods like linear classifier probing \cite{linearProbe} are also inapplicable: it requires not only numerous datapoints but also labels for them in order to train this classifier. In our case, we obtain only 1 set of tokens from training over the entire dataset. 

The only meaningful approach seems to be to compare the Aligner token with the LLaMA-Adapter tokens using t-SNE or directly. This could potentially reveal how our globally-shared token relates to the local tokens. We have conducted several comparisons and added the results to the main paper and the supplementary appendix. Here's a list of findings:
1. The Aligner 1 Token embedding is not necessarily the "average" of Aligner 10 Token or the LLaMA-Adapter, which has 300 tokens.
2. When we train the model with a different dataset, the embeddings of Aligner and those of the LLaMA-Adapter remain very close to each other. In fact, approximately half of the numbers in the embeddings are *exactly the same*, and the majority of the rest are still very close. This shows that very little change is needed to dramatically adapt the behaviors of an LLM.
3. The standard deviation of the gating factors in a layer is increasing as the layer goes to top, for both Aligner and LLaMA-Adapter. This may be in align with our general intuition that top layers generally requires larger adaptations. 

The figures are shown in section 4.4, figure 6 in the main paper and Appendix D in the supplement. We also put some analysis that doesn't yield any finding in Appendix D that you can take a look at. We would be pleased to perform any other suitable data analysis that you can recommend to us.