Paper: Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR-2024)

Reviewer comment (Experiments):
simple layernorm finetuning is reported to be including less trainable parameters and sensitive to a list of hyperparameters such as learning rate and training epoch, making it difficult to find the optimal performances in a quick way;

Author response:
>**Questions 2 and Weakness 1**: How to select the hyperparameters.

We have presented the parameter search procedure in Appendix A.1, where we searched for a range of learning rates for training the model. For the training epoch, we follow the standard practice of LLaVA to only train for **1 epoch** for both the pretraining stage and the fine-tuning stage **for all tuning methods**.

For applying LayerNorm tuning for new datasets and tasks, we recommend setting the learning rate to a higher value, as in our experiments, for LayerNorm tuning to perform well, the learning rate typically need to be set to 10x larger than LoRA tuning or full fine-tuning. We also present our explorations w.r.t. learning rate in the Appendix A.1, and **give our recommended searching range for both 7B and 13B models**.
We will include more detailed discussions in the final version of our paper.

Reviewer comment (Experiments):
not clear yet of how other researchers combine LayerNorm with other PEFT methods - are the results better or not?

Author response:
>**Questions 1 and Weakness 2**: Results of combining LayerNorm tuning with other PEFT.

We have provided the results of combining LayerNorm tuning with other PEFT in the response to Reviewer qMBg. The results of combined methods show an improvement from using only one of the tuning methods, indicating that LayerNorm is orthogonal to previous PEFT methods.

Reviewer comment (Experiments):
any further detailed results on combining LayerNorm tuning with other types of PEFT? such as those listed in https://github.com/huggingface/peft?

Author response:
>**Questions 1 and Weakness 2**: Results of combining LayerNorm tuning with other PEFT.

We have provided the results of combining LayerNorm tuning with other PEFT in the response to Reviewer qMBg. The results of combined methods show an improvement from using only one of the tuning methods, indicating that LayerNorm is orthogonal to previous PEFT methods.

Reviewer comment (Evaluation):
can you show some examples that tent to be better or worse when applying LayerNorm?

Author response:
>**Questions 3**: Cases of success and failure of applying LayerNorm Tuning.

In our paper, we have used the MME benchmark which contains the evaluation of MLLMs on both cognition and perception aspects, as shown in our Table 1 of the main paper. We can see that LayerNorm tuning can bring consistent improvements or at least match other tuning methods on the perception aspects. On the cognition aspect, the improvement is not consistent, and sometimes LayerNorm tuning falls behind other methods like LoRA tuning. However, note that LayerNorm tuning activates 10x less parameters than LoRA tuning, the performance gain from it is considered to be impressive.
We will include these discussions in the final version of our paper.

Reviewer comment (Experiments):
Lack of Exploration of Combined Training Strategies: The paper primarily focuses on the LayerNorm tuning strategy. A more comprehensive exploration, combining this with other training strategies, would provide a richer understanding of the method's potential and its interaction with other techniques. Such an exploration could offer more actionable insights for practitioners looking to adopt the proposed method.

Author response:
>**Questions 3 and Weakness 2**: Compare and combine with more PEFT baselines.

Thanks for the suggestion! We have performed the experiments of combining LayerNorm tuning with LoRA and LayerNorm tuning with Prefix-tuning. The results of MLLM with LLaMA-Chat-7B are presented in the table below.


|                           | MME/CS | MME/PS | MS-COCO |
|---------------------------|--------|--------|---------|
| LayerNorm tuning          | 651.3  | 219.3  | 75.34   |
| LoRA                      | 709.8  | 228.6  | 57.26   |
| Prefix                    | 541.0  | 202.1  | 50.78   |
| LayerNorm tuning + LoRA   | 734.8  | 245.6  | 75.61   |
| LayerNorm tuning + Prefix | 607.2  | 229.1  | 76.6    |


We can see that combining **LayerNorm tuning with other PEFT method brings additional improvement** in performance, indicating that the gain from LayerNorm tuning is orthogonal to the gains of other PEFT methods.
We will include these experiments in the final version of the paper.

Reviewer comment (Theory):
Empirical and Intuitive Explanations: The paper's explanations for the observed results lean heavily on empirical evidence and intuitive reasoning. A deeper theoretical analysis or a more rigorous exploration of the underlying mechanisms would strengthen the paper's claims and provide a more solid foundation for the proposed method.

Author response:
>**Questions 2 and Weakness 3**: Theoretical insights of LayerNorm tuning.

Our intuitions behind the success of LayerNorm tuning is provided in section 5, including the perspectives of domain adaptation, expressive power, and gradient variance. Apart from the empirical findings on the gradient variance in section 5.3, we have also provided theoretical proof in appendix A.2.2 showing that when the number of network parameters is larger as in the case of MLLMs, the gradient variance of LayerNorm will be smaller, providing a smoother optimization of the LayerNorm parameters. Thus giving such a significant result for parameter-efficient fine-tuning.

Reviewer comment (Evaluation):
In the experiments, there seems to be a performance variance between different LLMs, especially between the human-aligned 7B and 13B models. Could the authors elaborate on the potential reasons for this inconsistency?

Author response:
>**Questions 1 and Weakness 1**: Inconsistency across LLMs.

Comparing MM-LLaMA2-Chat-7B to MM-LLaAM2-Chat-13B, we can see that LayerNorm tuning on the 7B model gets a low score on the MME benchmark, but on the 13B model LayerNorm gets the best score on MME. Based on our analysis on the gradient variance of LayerNorm, we found that the larger the number of parameters, the smoother the gradient of LayerNorm will be, thus enabling a better optimization. The 13B model indeed contains more parameters than the 7B model, thus contributing to the better optimization of LayerNorm tuning, giving a better performance. Additionally, we would like to note that even at the 7B scale, our proposed LayerNorm tuning still gets the best performance on the VQAv2, MSCOCO, Flickr30k, and POPE benchmarks.

Reviewer comment (Theory):
While the paper provides empirical and intuitive explanations, could the authors shed light on any theoretical insights or hypotheses they might have about why tuning only the LayerNorm yields such significant results?

Author response:
>**Questions 2 and Weakness 3**: Theoretical insights of LayerNorm tuning.

Our intuitions behind the success of LayerNorm tuning is provided in section 5, including the perspectives of domain adaptation, expressive power, and gradient variance. Apart from the empirical findings on the gradient variance in section 5.3, we have also provided theoretical proof in appendix A.2.2 showing that when the number of network parameters is larger as in the case of MLLMs, the gradient variance of LayerNorm will be smaller, providing a smoother optimization of the LayerNorm parameters. Thus giving such a significant result for parameter-efficient fine-tuning.

Reviewer comment (Experiments):
Although existing experiments have demonstrated the effectiveness of the method, I suggest evaluating the model's capabilities comprehensively on more data, such as VSR, HM...

Author response:
We agree with the reviewer that more evaluations could provide a more holistic understanding of the effectiveness of LayerNorm tuning. However, in this paper, we focus more on the general abilities of MLLMs, thus we choose to evaluate the model on MME which is considered as a comprehensive evaluation benchmark for MLLMs. We have also provided the results of evaluating LayerNorm tuning on MMLU and TruthfulQA in the response to reviewer rJin to demonstrate the effectiveness of LayerNorm tuning.
We were not sure what VSR and HM stands for, we would gladly add the test on VSR and HM if the reviewer could point us to the resources for them.

Reviewer comment (Experiments):
Do we have any results comparing it with LoRA for non-multimodal tasks?

Author response:
We would like to thank the reviewer for their insightful and positive comment. 

>**Questions 1 and Weakness 1**: Results on non-multi-modal tasks.

We have performed experiment of LayerNorm tuning on the Platypus instruction tuning dataset (https://huggingface.co/datasets/garage-bAInd/Open-Platypus) and evaluation on MMLU and TruthfulQA which are both language only tasks.
The results are shown as follows, we use the LLaMA-2-7B-chat models for the experiments.

|                  | MMLU | TruthfulQA-mc1 |
|------------------|------|----------------|
| Fine-tuning      | 54.8 | 29.8           |
| LoRA             | 56.7 | 30.4           |
| LayerNorm tuning | 57.1 | 31.0           |

For LayerNorm tuning, we can see that it is **still effective on language-only tasks** compared to LoRA tuning and full fine-tuning, indicating the effectiveness of the proposed tuning paradigm.

Note that, since the size of Playtypus may be small for tuning a 7B model (*i.e.*, 25K training data, which is much smaller than the 80K multimodal data we employed for visual instruction tuning). The LoRA tuning is expected to be better than fine-tuning for the generation task under this situation [1].

[1] Revisiting Parameter-Efficient Tuning: Are We Really There Yet? EMNLP 2022