Paper: LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention (ICLR-2024)

Reviewer comment (Novelty):
The paper presents limited technical innovation and marginal performance improvement, as the concept of zero-initialized attention closely mirrors the zero-initialized convolution in ControlNet [Zhang et al., ICCV 2023], making it appear more like an engineering design rather than a groundbreaking technical contribution.

Author response:
Sorry for the confusion caused. Our zero-initialized attention shares similar high-level motivations with ControlNet, i.e., progressively injecting new knowledge into the pre-trained model. However, our method is very different in four aspects.

1. **Target Task.**  ControlNet aims to inject visual conditions into pre-trained diffusion models, which conducts text-to-image generation. Our target is to incorporate pre-trained LLMs with new instruction signals (language or multi-modality), tuning an LLM into an instruction-following model.
2. **Methodology.** On top of the CNN-based U-Net, ControlNet zero-initializes ***traditional convolutional layers***, and ***directly adds*** new conditions with the pre-trained features by residual connections. In contrast, considering the transformer architecture of LLMs, we propose to achieve zero-initialization within the self-attention layer by ***a learnable gating factor***. Then, we inject the new instruction cues into LLMs ***naturally through the attention process***, which adaptively controls the interaction between prompts and word tokens.
3. **Parameter Efficiency.** ControlNet creates an entire copy of U-Net's encoder for training, which contains over 300M learnable parameters. However, our method only contains 1.2M parameters for adaption prompts and zero-initialized gating factors (1.8M for the multi-modal variant),  indicating superior parameter efficiency.

Therefore, the zero-initialized attention of LLaMA-Adapter is distinct from ControlNet in both ideas and technical details.

Reviewer comment (Novelty):
Of the four main characteristics highlighted by the authors in the introduction, both (1) parameter efficiency and (3) plugin with expertise are attributes already provided by LoRA.

Author response:
It is true that Alpaca-LoRA can also achieve parameter- and time-efficiency (4.2M, 1.5h), even though our LLaMA-Adapter is more efficient (1.2M, 1h). More importantly, the main difference of our approach is ***the generalization capacity for multi-modal input***. LoRA appends trainable rank decomposition matrices into each transformer layer, which entirely inherits the original structures of LLMs. Therefore, LoRA cannot be extended to process new input features. In contrast, our LLaMA-Adapter with adaption prompts and zero-initialized attention is generalizable to accept new input conditions for instruction following, e.g., the plugin with visual reasoning expertise. With our approach, LLaMA can be fine-tuned into a multi-modal LLM with competitive visual understanding capability to existing multi-modal LLMs (shown in ***Table 3 of the paper***), which indicates our promising generalizability for wider application scenarios.

Reviewer comment (Evaluation):
Regarding (2) training efficiency, the improvement over LoRA is incremental, reducing training time from 1.5 hours to 1 hour (in Table 1) while achieving comparable accuracy (in Figure 5).

Author response:
It is true that Alpaca-LoRA can also achieve parameter- and time-efficiency (4.2M, 1.5h), even though our LLaMA-Adapter is more efficient (1.2M, 1h). More importantly, the main difference of our approach is ***the generalization capacity for multi-modal input***. LoRA appends trainable rank decomposition matrices into each transformer layer, which entirely inherits the original structures of LLMs. Therefore, LoRA cannot be extended to process new input features. In contrast, our LLaMA-Adapter with adaption prompts and zero-initialized attention is generalizable to accept new input conditions for instruction following, e.g., the plugin with visual reasoning expertise. With our approach, LLaMA can be fine-tuned into a multi-modal LLM with competitive visual understanding capability to existing multi-modal LLMs (shown in ***Table 3 of the paper***), which indicates our promising generalizability for wider application scenarios.

Reviewer comment (Experiments):
Reducing the rank of LoRA can further reduce the number of trainable parameters and potentially speed up training; the authors should include an additional ablation study on this.

Author response:
Thanks for your advice! It is very necessary to evaluate the performance of Alpaca-LoRA with fewer parameters comparable to LLaMA-Adapter. The default rank of Alpaca-LoRA is 8, and we respectively show the results with the ranks of 2, 4, and 16 in the table below. For a more comprehensive assessment, we evaluate their language processing capabilities on **Open LLM benchmark** [1]. It evaluates LLMs' generative abilities in four different tasks: AI2 Reasoning Challenge [2], HellaSwag [3], MMLU [4], and TruthfulQA [5]. Each task contains challenging data samples over a wide range of knowledge domains. 
| Model | Rank | Param | Time | Avg | ARC | HellaSwag | MMLU | TruthfulQA |
|---|---|---|---|---|---|---|---|---|
| Alpaca-LoRA | 2 | 1.0 | 1.48 | 50.9 | 53.6 | 77.9 | 37.9 | 34.0 |
| Alpaca-LoRA | 4 | 2.1 | 1.49 | 50.8 | 53.5 | 77.8 | 37.5 | 34.4 |
| Alpaca-LoRA | 8 | 4.2 | 1.49 | 50.7 | 53.2 | 78.1 | 37.1 | 34.5 |
| Alpaca-LoRA | 16 | 8.4 | 1.5 | 50.8 | 53.0 | 78.0 | 37.1 | 34.9 |
| LLaMA-Adapter | - | 1.2 | 1 | 52.2 | 54.7 | 78.8 | 34.9 | 40.4 |

As shown above, lower ranks of LoRA can effectively reduce the learnable parameters from 8.4M to 1.0M, and slightly lower the training time from 1.5h to 1.48h. However, our LLaMA-Adapter with 1.2M parameters and 1h still attains the best average result, demonstrating a good trade-off between performance and training efficiency.

Reviewer comment (Experiments):
The authors claim that "LLaMA-Adapter enables fine-tuning of large-scale language models on mobile devices," but this lacks empirical support; while LLaMA-Adapter may reduce memory usage, it does not necessarily reduce computation, which could still be a constraint on mobile devices.

Author response:
Thanks for pointing out! Our claim concerning mobile devices is not rigorously precise. We have rectified it in the revised paper as "LLaMA-Adapter enables more efficient storage of large-scale language models on mobile devices".

Reviewer comment (Experiments):
The learnable adaptation tokens have been prefixed to the last 30 of 32 layers; what might the implications be if applied across all transformer layers?

Author response:
As you suggested, we conduct an ablation study on the insertion layer numbers on ScienceQA in the table below. As shown, inserting at the ***last 30 layers*** is better than the ***early 30 layers*** or ***all layers***. 
This indicates that it's more effective to inject the instructional cues into layers capturing higher-level semantics for lanuguage generation. There also exists an optimal insertion number from the higher layers, since too few layers might underfit the training data, while too many layers would adversely disturb the early encoding of input words. If one has limited resources to identify the best insertion layer number, simply inserting our modules into all transformer layers is generally a good solution.
| Layers| Params|Val Acc.
|:---|:---:|:---
|Last 30|1.79|83.85
|Early 30|1.79|79.69
|All 32|1.83|81.03

Reviewer comment (Experiments):
I think a number of baselines and performance of the proposed method on multi-modal evaluation are weak to convince the proposed method is more effective than full finetuning LLMs or other efficient methods. I understand there are a lot of recent multi-modal models but many of them are concurrent works. However, I think BLIP2 [1] can be treated as a baseline, which is also a prior work of reported baseline, mini-GPT4. Based on this, I think the reported performances in Table 3 are weak compared to others. For example, the proposed method achieved a 973 score on the MME benchmark, while BLIP2 did 1293. Furthermore, a recent multi-modal model LLaVA-1.5 [2], which fully fine-tunes LLMs, achieved the 1510 score using the Vicuna 7B model. Therefore, I think this manuscript should include empirical backups to show the proposed method is still effective and efficient in the multi-modal domain.

Author response:
> **Q1: Treating BLIP-2 as the baseline to LLaMA-Adapter for multi-modal evaluation**

>
Thanks for your advice! BLIP-2 is a good and widely adopted baseline for existing multi-modal LLMs. However, please note that it is ***not appropriate*** to judge the capability of our approach solely based on BLIP-2's MME performance. The two reasons are as follows.

1. **Targeting on Different Capabilities.** Our LLaMA-Adapter, along with the concurrent LLaVA and MiniGPT-4, all aim to acquire the ***visual instruction-following capacity***, namely, following human instructions to generate human-like answers conditioned on image input. In contrast, BLIP-2 focuses on the ***fundamental vision-language understanding capability***. Therefore, the former visual instruction models tend to output long and detailed sentences like human conversations of daily life, while the latter BLIP-2 prefers to generate accurate answers with concise language.

2. **Evaluation Bias in MME.** MME mainly evaluates the fundamental perception and cognition abilities of multi-modal LLMs, for which the ground-truth answer contains only one word: "Yes" or "No". As analyzed above, such short-answer evaluation is more advantageous to BLIP-2 than visual instruction models (LLaMA-Adapter, LLaVA, and MiniGPT-4). For example, MiniGPT-4 is instruction-tuned based on the pre-trained vision encoder and Q-Former of BLIP-2, while performing worse than BLIP-2 in MME, as shown in the table below. In addition, InstructBLIP [1] further conducts instruction tuning upon the pre-trained BLIP-2, and also underperforms BLIP-2 on MME.
| Method | MME |  |  | MMBench |  |  |  |  |  |  | MM-Vet |  |  |  |  |  |  |
|---|:-:|---|---|:-:|---|---|---|---|---|---|:-:|---|---|---|---|---|---|
|  | **All** | P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** |Rec | OCR | Know | Gen | Spat | Math |
| BLIP2 | 1584 | 1294 | 290 | - | - | - | - | - | - | - | 22.4 |27.5 | 11.1 | 11.8 | 7.0 | 16.2 | 5.8 |
| InstructBLIP | 1505 | 1213 | 292 | 33.9 | 21.6 | 47.4 | 22.5 | 33.0 | 24.4 | 41.1 | 26.2 |32.4 | 14.6 | 16.5 | 18.2 | 18.6 | 7.7 |
| Mini-GPT4 | 1159 | 867 | 292 | 23.0 | 13.6 | 32.9 | 8.9 | 28.7 | 11.2 | 28.3 | 22.1 |27.5 | 11.1 | 11.8 | 7.0 | 16.2 | 5.8 |
| LLaVA | 718 | 503 | 215 | 36.2 | 15.9 | 53.6 | 28.6 | 41.8 | 20.0 | 40.4 | 23.8 | 28.0 | 17.1 | 16.3 | 18.9 | 21.2 | 11.5 |
| LLaMA-Adapter | 1222 | 973 | 249 | 39.5 | 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 | 32.8 |38.5 | 20.3 | 31.4 | 33.4 | 22.9 | 3.8 |


Therefore, ***the instruction-following ability of visual instruction models***, e.g., LLaMA-Adapter, ***cannot be entirely reflected by MME scores***. Instead, on other multi-modal benchmarks, e.g., MMBench [2] and MM-Vet [3], our approach exhibits significant advantages over BLIP-2 or BLIP-2-based instruction models (InstructBLIP and MiniGPT-4):

1. **MMBench** [2] comprehensively measures 20 vision-language reasoning tasks with three levels of ability dimensions. MMBench introduces a new CircularEval strategy by using ChatGPT, which converts free-form predictions into pre-defined choices to alleviate the bias for BLIP-2. As shown in the table above, our LLaMA-Adapter showcases a clear advantage to BLIP-2-based instruction models.

2. **MM-Vet** [3] examines 6 core vision-language capabilities with 16 kinds of integration. It introduces an LLM-based evaluator for open-ended outputs, which thoroughly measures the answering performance of visual instruction models. As shown in the table above, our LLaMA-Adapter also performs much better than BLIP-2.

#### Reference:
#### [1] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv 2023.
#### [2] MMBench: Is Your Multi-modal Model an All-around Player?. arXiv 2023.
#### [3] MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv 2023.

Reviewer comment (Experiments):
I think the most important baselines on instruction following evaluation are Alpaca and Alpaca-LoRa. However, the manuscript only provides qualitative comparisons and brief comparisons in Figure 2. I think it would be great if more quantitative comparisons could be included in the manuscript.

Author response:
> **Q3:  More quantitative comparisons with Alpaca and Alpaca-LoRA**

>
Thanks for your advice! Besides qualitative results, we have compared them on the GPT-4 evaluation benchmark [6] in ***Figure 5 of the paper***, which utilizes GPT-4 to assess the response quality on 80 questions. As you suggested, we further evaluate the language processing capacity of the three methods on ***Open LLM benchmark*** [7]. It evaluates LLMs' generative abilities in four different tasks: AI2 Reasoning Challenge [8], HellaSwag [9], MMLU [10], and TruthfulQA [11]. Each task contains challenging data samples over a wide range of knowledge domains.

As shown in the table below, LLaMA-Adapter still achieves the best average performance than Alpaca's full fine-tuning and Alpaca-LoRA. This demonstrates the strong language instruction-following ability of our approach.
| Method | Avg | ARC | HellaSwag | MMLU | TruthfulQA |
|---|:-:|:-:|:-:|:-:|:-:|
| Alpaca | 49.2 | 49.1 | 77.7 | 33.8 | 36.3 |
| Alpaca-LoRA | 50.7 | 53.0 | 77.9 | 37.1 | 34.9 |
| LLaMA-Adapter | 52.2 | 54.7 | 78.8 | 34.9 | 40.4 |

#### Reference:
#### [4] Improved Baselines with Visual Instruction Tuning. arXiv 2023.
#### [5] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.
#### [6] https://lmsys.org/blog/2023-03-30-vicuna/.
#### [7] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/.
#### [8] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv 2018.
#### [9] HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019.
#### [10] Measuring Massive Multitask Language Understanding. ICLR 2021.
#### [11] TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.

Reviewer comment (Experiments):
The proposed method has a sensitive hyperparameter of "a number of insertion layers". In particular, Section 3.2 emphasizes the risk of early insertion layers, while the ablation study in Section 4.3 emphasizes the importance of increasing the number of insertion layers. I think the manuscript could provide more explanation on the choice of the hyperparameter.

Author response:
> **Q4: More explanation on the choice of the "number of insertion layers"**

>
Sorry for the confusion caused. Inserting prompts with zero-initialized attention into higher transformer layers is better than earlier layers. This is because it's more effective to inject the instructional cues into layers capturing higher-level semantics for language generation. There also exists an optimal insertion layer number, since too few layers might underfit the training data, while too many layers would adversely disturb the early encoding of input words.

We conduct an ablation study for the insertion layer numbers on ScienceQA in the table below. As shown, inserting at the ***last 30 layers*** is better than the ***early 30 layers*** or ***all 32 layers***. Also, if one has limited resources to identify the best insertion layer number, simply inserting our modules into all transformer layers would generally be a good solution.
| Layers| Params|Val Acc.
|:---|:---:|:---
|Last 30|1.79|83.85
|Early 30|1.79|79.69
|All 32|1.83|81.03

Reviewer comment (Experiments):
The proposed method has few learnable parameters. So, I am curious about the possibility of learning large amounts of data or a large number of tasks.

Author response:
> **Q6: The method has few learnable parameters. How about the possibility of learning large-scale data or tasks?**

>
Thanks for your advice! It is very necessary to evaluate the scalability of our LLaMA-Adapter for more data and tasks.

1. ***More Instruction Data.*** By default, we utilize a combination of Alpaca's data (52K) and LLaVA-I (158K) for visual instruction tuning. Here, we progressively add more question-answering data to enlarge the instruction datasets of LLaMA-Adapter: the sampled **83K VQAv2** by LLaVA-1.5 and the entire **204K VQAv2**. As shown in the table below, the increasing instruction tuning data contributes to better multi-modal reasoning results on all benchmarks, demonstrating our method’s favorable scalability to a larger amount of data.
| Data Size | MME ||  | MMBench |  |  |  |  |  |  | LVLM-eHub |  |  |  |  |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| LLaMA-Adapter | **All** |P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** | VP | VKA | VR | VC |
| Alpaca+LLaVA-I |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.67| 0.81 | 0.44 | 0.83 | 0.59 |
| +VQAv2 (83K) |1256| 1007 | 249 |43.4| 22.9 | 44.7 | 31.3 | 46.7 | 46.9 | 50.3 |0.69| 0.84 | 0.42 | 0.88 | 0.63 |
| +VQAv2 (204K) |1618| 1272 | 346 |60.1| 34.7 | 65.3 | 48.7 | 63.1 | 57.3 | 69.3 |0.72| 0.86 | 0.44 | 0.92 | 0.65 |

2. ***More Learning Tasks.*** As shown in the paper and rebuttal, our LLaMA-Adapter has superior generalization capacity over a wide range of tasks, including ***language instruction following***, e.g., question answering and code generation, ***multi-modal reasoning tasks***, e.g., visual question answering and image captioning, and ***fine-tuning conventional vision or language models***. For example, in the table below, we report the specific results on the task of optical character recognition (***OCR***) on several benchmarks: DocVQA [12], TextVQA [13], and OCR-VQA [14]. As shown, our LLaMA-Adapter achieves competitive performance compared to other methods. The experiments indicate that, given a small number of learnable parameters, our approach is generalizable to comprehend diverse vision-language tasks.
| Datasets | LLaMA-Adapter | LLaVA | MiniGPT-4 | BLIP2 |
|---|:---:|:---:|:---:|:---:|
| DocVQA | 8.13 | 6.26 | 2.65 | 4.75 | 
| TextVQA | 43.76 | 38.92 | 19.40 | 31.98 | 
| OCR-VQA | 38.12 | 23.40 | 16.85 | 38.85 | 

#### Reference:
#### [12] DocVQA: A Dataset for VQA on Document Images. WACV 2021.
#### [13] Towards VQA Models That Can Read. CVPR 2019.
#### [14] OCR-VQA: Visual Question Answering by Reading Text in Images. ICDAR 2019.

Reviewer comment (Experiments):
I would say the comparison is rather limited, newer and better base LLMs are available, adding comparison or applying the proposed method to models such as MPT, Falcon, or LLaMA-v2 could make the paper stronger.

Author response:
> **Q1: Applying the method to newer LLMs, e.g., MPT, Falcon, and LLaMA-2, could make the paper stronger**

>
Thanks for your advice! Due to the limited time period of rebuttal, we apply our LLaMA-Adapter (including the adaption prompts and zero-initialized attention) to the latest LLM: ***LLaMA-2-7B*** [1]. LLaMA-2 is a stronger LLM that outperforms LLaMA on a wide range of tasks. For a more comprehensive evaluation, we tune the multi-modal variant of our approach, and evaluate it on three widely-adopted benchmarks: MME, MMBench, and LVLM-eHub. As shown in the table below, our approach can achieve higher visual instruction-following results with the stronger LLaMA-2. This demonstrates the superior generalization capacity of our method for instruction-tuning different pre-trained LLMs.
| LLM Model|MME ||  | MMBench |  |  |  |  |  |  | LVLM-eHub |  |  |  |  |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| LLaMA-Adapter | **All** |P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** | VP | VKA | VR | VC |
| w LLaMA-7B|1222| 973 | 249 | 39.5 | 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 | 0.67| 0.81 | 0.44 | 0.83 | 0.59 |
| w LLaMA-2-7B |1267|968|299|42.1|15.9|48.2|23.6|48.9|35.1|51.6|0.68|0.83|0.43|0.86|0.61|

Reviewer comment (Evaluation):
Evaluations are a bit narrow, I would recommend adding things like counterfactual reasoning [R1,R2] or object hallincations [R3].

Author response:
> **Q2: It's better to add more evaluations, e.g., counterfactual reasoning [2] or object hallucinations [3]**

>
Thanks for your advice! Besides the regular evaluation in ***Table 3 of the paper***, it is very necessary to explore the counterfactual reasoning abilities and object hallucination issues of multi-modal LLMs. As you suggested, we respectively evaluate our multi-modal LLaMA-Adapter on two suggested benchmarks as follows.

1. **C-VQA** [2] for counterfactual reasoning. C-VQA contains 2k counterfactual question and answer pairs, which are collected from VQAv2 [3] and supplemented by ChatGPT. As shown in the table below, for three groups of questions, LLaMA-Adapter performs comparably to the concurrent LLaVA. Especially for the ***Numerical indirect*** questions, our approach achieves the best counterfactual reasoning results (34.3) and the least performance loss (5.6$\downarrow$) than all other models.
Method|Numerical direct$\uparrow$ (Loss$\downarrow$)|Numerical indirect$\uparrow$ (Loss$\downarrow$)|Boolean$\uparrow$ (Loss$\downarrow$)
-|:-:|:-:|:-:
ViperGPT|80.6 (19.4$\downarrow$)|31.6 (68.4$\downarrow$)|21.6 (72.4$\downarrow$)
LLaVA-7B|27.0 (9.9$\downarrow$)|25.0 (15.2$\downarrow$)|58.5 (4.8$\downarrow$)
LLaVA-13B|24.8 (11.9$\downarrow$)|20.8 (21.2$\downarrow$)|56.3 (4.7$\downarrow$)
LLaMA-Adapter-7B|30.1 (5.8$\downarrow$)|34.3 (5.6$\downarrow$)|45.8 (14.5$\downarrow$)

2. ***POPE*** [4] for object hallucinations. POPE evaluates object hallucinations as a binary classification task, and includes 500 images from MSCOCO with 6 questions per sample. As shown in the table below, for different evaluation settings, LLaMA-Adapter with LLaMA-7B attains competitive accuracy compared to other multi-modal LLMs with LLaMA-13B, which indicates our relatively stronger robustness to object hallucination problems.
Method|Random|Popular|Adversarial
-|:-:|:-:|:-:
mPLUG-Owl-7B|53.30|50.63|50.67
LLaVA-13B|54.43|52.43|50.77
MM-GPT-7B|50.03|50.00|50.00
LLaMA-Adapter-7B|75.47|60.43|60.66

#### Reference:
#### [1] Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv 2023.
#### [2] What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models. arXiv 2023.
#### [3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.
#### [4] Evaluating Object Hallucination in Large Vision-Language Models. arXiv 2023.

Reviewer comment (Experiments):
What would happen if the base model is larger? Does the proposed method scales with the base model size?

Author response:
> **Q3: It's better to test the scalability of the method, e.g., using a larger base model or more instruction tuning data**

>
Thanks for your suggestion! As you suggested, we evaluate the scalability of our LLaMA-Adapter from two aspects: base model size and instruction data size.

1. **Base Model Size.** In the original paper, we conduct all experiments with LLaMA-7B. To scale up the base model, we further implement LLaMA-Adapter with a larger LLaMA-13B, and compare the results on three benchmarks as follows. On most tasks, LLaMA-13B attains better performance than LLaMA-7B, which demonstrates that our approach can be flexibly adapted to larger LLMs for potentially better multi-modal instruction-following capacity.
| LLM Size|MME ||  | MMBench |  |  |  |  |  |  | LVLM-eHub |  |  |  |  |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| LLaMA-Adapter | **All** |P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** | VP | VKA | VR | VC |
| w LLaMA-7B |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.67| 0.81 | 0.44 | 0.83 | 0.59 |
| w LLaMA-13B |1299|1021|278|40.1|13.9|47.4|23.7|46.1|34.2|51.2|0.64|0.82|0.39|0.85|0.50|

Reviewer comment (Experiments):
Also, Does the proposed method scales with the number of instruction data used to tune the model?

Author response:
> **Q3: It's better to test the scalability of the method, e.g., using a larger base model or more instruction tuning data**

>
Thanks for your suggestion! As you suggested, we evaluate the scalability of our LLaMA-Adapter from two aspects: base model size and instruction data size.

1. **Base Model Size.** In the original paper, we conduct all experiments with LLaMA-7B. To scale up the base model, we further implement LLaMA-Adapter with a larger LLaMA-13B, and compare the results on three benchmarks as follows. On most tasks, LLaMA-13B attains better performance than LLaMA-7B, which demonstrates that our approach can be flexibly adapted to larger LLMs for potentially better multi-modal instruction-following capacity.
| LLM Size|MME ||  | MMBench |  |  |  |  |  |  | LVLM-eHub |  |  |  |  |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| LLaMA-Adapter | **All** |P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** | VP | VKA | VR | VC |
| w LLaMA-7B |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.67| 0.81 | 0.44 | 0.83 | 0.59 |
| w LLaMA-13B |1299|1021|278|40.1|13.9|47.4|23.7|46.1|34.2|51.2|0.64|0.82|0.39|0.85|0.50|


2. **Instruction Data Size.** By default, we utilize a combination of Alpaca's data (52K) and LLaVA-I (158K) for visual instruction tuning. Here, we progressively add more question-answering data to enlarge the instruction datasets of LLaMA-Adapter: the sampled ***83K VQAv2*** by LLaVA-1.5 [5] and the entire ***204K VQAv2***. As shown in the table below, the increasing instruction tuning data leads to better multi-modal reasoning results on three benchmarks, demonstrating our method’s favorable scalability to data size.
| Data Size | MME ||  | MMBench |  |  |  |  |  |  | LVLM-eHub |  |  |  |  |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| LLaMA-Adapter | **All** |P | C | **All** | LR | AR | RR | FP-S | FP-C | CP | **All** | VP | VKA | VR | VC |
| Alpaca+LLaVA-I |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.6675| 0.81 | 0.44 | 0.83 | 0.59 |
| +VQAv2 (83K) |1256| 1007 | 249 |43.4| 22.9 | 44.7 | 31.3 | 46.7 | 46.9 | 50.3 |0.6925| 0.84 | 0.42 | 0.88 | 0.63 |
| +VQAv2 (204K) |1618| 1272 | 346 |60.1| 34.7 | 65.3 | 48.7 | 63.1 | 57.3 | 69.3 |0.7175| 0.86 | 0.44 | 0.92 | 0.65 |


#### Reference:
#### [5] Improved Baselines with Visual Instruction Tuning. arXiv 2023.