Paper: Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE (ICLR-2024)

Reviewer comment (Experiments):
The work focuses on SFT, whether it can be generalized to pre-training on massive data.

Author response:
> ***1. The work focuses on SFT, and whether it can be generalized to pre-training on massive data?***

  - Generalizing to pre-training on massive data is not our focus. Our main motivation involves adding PEFT techniques (such as our LoRA-MoE) to the existing MLLMs to achieve better downstream performance generalization on more tasks and more modalities. 

  - Considering that other 2D MLLMs methods (except KOSMOS series, but they face replication problems) have also not mentioned training on massive datasets, for a fair comparison, we will not try to address this issue recently.

  - Besides, collecting massive datasets for some modalities, such as point clouds, is quite challenging, thus preventing us from pre-training MLLMs on massive data in point cloud modality.

Reviewer comment (Experiments):
Lack of comparison with some classic and latest methods, such as MiniGPT4, mPLUG-Owl, LLAVA-1.5, Qianwen-VL, etc.

Author response:
> ***2. Lack of comparison with some classic and latest methods, such as MiniGPT4, mPLUG-Owl, LLAVA-1.5, Qianwen-VL, etc.***

  - Our focus is not on proposing an entirely new SOTA 2D MLLM, but rather on addressing potential tug-of-war issues in downstream tasks by parameter-efficient fine-tuning existing MLLMs (such as LoRA). Therefore, direct comparison with the latest 2D MLLMs does not align with our motivation. 

  - For instance, as mentioned by Reviewer 3GST, InstructBLIP outperforms our baseline + LoRA-MoE on captioning/VQA tasks. It can only indicate that the baseline method used by InstructBLIP (BLIP-2) is much stronger than our chosen baseline method (LLaVA and LAMM). Additionally, such comparisons are completely unrelated to the motivation of our paper, because they cannot reflect improvements in task interference that our LoRA-MoE brings to the baseline model. Comparing the improvements with the baseline method itself on multi-task and multi-modal setup is more important, rather than comparing with other MLLMs. Therefore, we only compare the baseline model with LoRA and LoRA-MoE to explore the effectiveness of LoRA-MoE in this paper. 

  - A further discussion on this issue is why not use stronger baseline methods like InstructBLIP, KOSMOS series, and Shikra? It is because only LLaVA and LAMM are feasible options for us to adopt LoRA-MoE among existing 2D MLLM methods at that time.
    
    - LLaVA-1.5 was released as a preprint on 10/5, exceeding the submission deadline of ICLR.
    
    - KOSMOS series, InstructBLIP, and Qianwen-VL do not release their entire training datasets, so they face replication problems.
    
    - Shikra mainly focuses on 2D grounding tasks. Although Shikra is capable of performing other tasks such as captioning and VQA, it does not conduct a zero-shot evaluation on captioning/VQA tasks (Flicker30k and VQAv2). If we directly remove related datasets (Flickr30k, VQAv2) from Shikra's training set for zero-shot evaluation, we cannot guarantee the zero-shot performance of Shikra on captioning/VQA tasks.

    - Other 2D MLLM methods (like MiniGPT4, mPLUG-owl, etc.) do not significantly outperform LLaVA and LAMM. Considering the simplicity of LLaVA and LAMM, and both of them provide implementation with LoRA, we choose these two methods as our baseline.

  - Besides, even if we ignore the motivation discussed above, all of the mentioned methods, including but not limited to MiniGPT4, mPLUG-Owl, LLAVA-1.5, Qianwen-VL, are trained on different datasets, different training strategies (e.g., PEFT vs. full-parameters FT, one-stage vs. two-stage), different pre-training weights (e.g., Vicuna-13b vs. Vicuna-7b, Vicuna vs. LLaMA vs. BLIP-2). Especially the differences  in datasets have a significant impact on the performance of MLLMs. Considering that our focus is not on dataset design but on model design, directly comparing their performance is unfair.

  - Although such comparisons are unfair and do not align with our motivation, we still provide comparisons with some latest methods in 2D modality in Table E. An apparent example of unfairness is that many models in the table fail in the detection evaluation. Although their training sets contain COCO, most of them mainly focus on captioning tasks, and the localization capability is not sufficiently trained, so they achieve poor results on VOC. (Note that **Table A** is in the next comment.)

Reviewer comment (Evaluation):
In Figure 5, Loras2 has a probability of 0 on 6 tasks. What could be the possible reasons for this phenomenon? Could you provide more visualizations of experts and provide some insightful analysis?

Author response:
>***4. In Figure 5, Loras2 has a probability of 0 on 6 tasks. What could be the possible reasons for this phenomenon?***
  - Although LoRA#2 is not used in all 2D tasks (Figure 2), it is essential in the learning. We conducted experiments using three experts, as shown in **Table C**, and the performance is worse. It indicates that #2 is used in the middle-early stages of learning, but after convergence, the gate does not select #2 due to load balancing issues. We will add this table to the final version.
    
    **Table C**
    | Gate | #Experts | VOC Recall@ 0.5 | VOC Prec@ 0.5 | ScienceQA Acc@1 |
    | --- | --- | --- | --- | --- |
    |  Top-2 | 1 | 7.61 | 5.95 | 40.31 |
    |  Top-2 | 3 | 21.64 | 12.20 | 44.72 |
    |  Top-2 | 4 | 39.04 | 35.21 | 46.95 |
  - #2 is not used due to the load balancing issue. As mentioned in the Ablation, the sample-based gate is incompatible with the load balancing strategy. For example, it is more reasonable to assign detection samples to a LoRA expert proficient in localizing than the other experts for the purpose of load balancing. Additionally, the experimental results in Table 6 confirm this viewpoint.

Reviewer comment (Experiments):
Can you analyze the impact of k in the top-k sparse gate?

Author response:
>***6. Can you analyze the impact of k in the top-k sparse gate?***
  - We provide results of adopting different top-k gates in **Table D**. We will add this table to the final version.
    
    **Table D**
    | Gate | #Experts | VOC Recall@ 0.5 | VOC Prec@ 0.5 | ScienceQA Acc@1 |
    | --- | --- | --- | --- | --- |
    | - | 1 | 7.61 | 5.95 | 40.31 |
    | top-1 | 4 | 22.42 | 21.23 | 36.88 |
    | top-2 | 4 | 39.04 | 35.21 | 46.95 |
    | top-3 | 4 | 38.57 | 36.02 | 43.89 |
    - The top-1 gate has poor performance due to its less flexibility in choice (only 4 combinations) compared with the top-2 and top-3 gate. While in the top-3 gate scenario, we find that the third choice is relatively small in most tasks (e.g., in classification tasks, the routing weight is like [0.1, 0., 0.4, 0.5]), therefore it is very similar to the top-2 gate, and achieves similar results as the top-2 gate.

Reviewer comment (Experiments):
The authors demonstrate the existence of the tug-of-war problem simply by optimizing 2D detection and VQA simultaneously.

Author response:
* First, in the pilot study, we demonstrate the existence of the tug-of-war on the whole LAMM-v2 datasets, including detection, VQA, caption, and classification tasks, not only optimizing detection and VQA.

* We reorganize the related results of multimodal tug-of-war issues (in Table 4) in **Table A**.

  **Table A**
  | FT. dataset | MoE | 2D Caption (ZS. Flickr30k) | 3D Caption (FT. Scan) | 3D Caption (ZS. NR3D) | Avg. |
  | --- | --- | --- | --- | --- | --- |
  | 2D | X | 0.21 | - | - | - |
  | 3D | X | - | 35.1 | 16.19 | - |
  | 2D + 3D | X | 0.04 | 19.76 | 8.26 | - |
  | 2D + 3D | O | 10.06 | 33.29 | 17.22 | 43.91% ⬆️ |

  * From this table, we can find that the tug-of-war issues also exist in multimodal setup, and can even be more severe. Here we primarily compare 2D and 3D captioning tasks. When we introduce more modalities into instruction tuning, we can observe a huge performance drop, especially in 3D captioning tasks (35 -> 19, 16 -> 8). After applying LoRA-MoE, we can find that the performance of 3D captioning tasks can reach the level of performance that fine-tuned on the single 3D modality. Meanwhile, the performance of 2D captioning is also greatly improved.

* This is a good perspective for analyzing the tug-of-war problems in the multimodal setup. We will add the tables and analysis to the final version. Thanks for your advice.

Reviewer comment (Experiments):
Considering the authors also include multi-modality, it would be better to include preliminary experiments also in this direction (e.g., 2D & 3D captioning).

Author response:
* First, in the pilot study, we demonstrate the existence of the tug-of-war on the whole LAMM-v2 datasets, including detection, VQA, caption, and classification tasks, not only optimizing detection and VQA.

* We reorganize the related results of multimodal tug-of-war issues (in Table 4) in **Table A**.

  **Table A**
  | FT. dataset | MoE | 2D Caption (ZS. Flickr30k) | 3D Caption (FT. Scan) | 3D Caption (ZS. NR3D) | Avg. |
  | --- | --- | --- | --- | --- | --- |
  | 2D | X | 0.21 | - | - | - |
  | 3D | X | - | 35.1 | 16.19 | - |
  | 2D + 3D | X | 0.04 | 19.76 | 8.26 | - |
  | 2D + 3D | O | 10.06 | 33.29 | 17.22 | 43.91% ⬆️ |

  * From this table, we can find that the tug-of-war issues also exist in multimodal setup, and can even be more severe. Here we primarily compare 2D and 3D captioning tasks. When we introduce more modalities into instruction tuning, we can observe a huge performance drop, especially in 3D captioning tasks (35 -> 19, 16 -> 8). After applying LoRA-MoE, we can find that the performance of 3D captioning tasks can reach the level of performance that fine-tuned on the single 3D modality. Meanwhile, the performance of 2D captioning is also greatly improved.

* This is a good perspective for analyzing the tug-of-war problems in the multimodal setup. We will add the tables and analysis to the final version. Thanks for your advice.

Reviewer comment (Reproducibility):
As shown in Fig. 2, the router input consists of the system prompt, modality embedding, and the question embedding, which contradicts with Tab. 5. Clarification is needed on how exactly to construct the router input.

Author response:
* We construct the router input following Equation 3, i.e., use question embedding only. And we obtain the question embedding on-the-fly.

* We admit that Figure 2 is a little confusing. [Here](https://github.com/iclrxoct/iclr2024xoct/blob/main/3GST_1.png) is the updated version. We will modify it in the final version. Thanks for your advice.

Reviewer comment (Reproducibility):
Clarification is needed on how to get the question embedding—whether the authors utilize another embedding model or generate it on-the-fly.

Author response:
* We construct the router input following Equation 3, i.e., use question embedding only. And we obtain the question embedding on-the-fly.

* We admit that Figure 2 is a little confusing. [Here](https://github.com/iclrxoct/iclr2024xoct/blob/main/3GST_1.png) is the updated version. We will modify it in the final version. Thanks for your advice.

Reviewer comment (Reproducibility):
It is unclear whether the authors adopt LoRA-MoE in each Transformer block.

Author response:
* In LoRA-MoE, LoRA experts are adopted in each Transformer block and each q/k/v/output projection layer. Specifically, we implement LoRA-MoE based on the original PEFT repository, so we replace all LoRA modules in LLMs with LoRA-MoE.

* We will add these implementation details in the final version. Thanks for your advice.

Reviewer comment (Reproducibility):
It is unclear whether the authors adopt LoRA only for the MLP, similarly to the original MoE.

Author response:
* In LoRA-MoE, LoRA experts are adopted in each Transformer block and each q/k/v/output projection layer. Specifically, we implement LoRA-MoE based on the original PEFT repository, so we replace all LoRA modules in LLMs with LoRA-MoE.

* We will add these implementation details in the final version. Thanks for your advice.

Reviewer comment (Experiments):
According to Sec. 4.1, the zero-shot evaluation conducted in this paper is only about zero-shot evaluation on novel datasets of the training tasks (e.g., ScienceQA for evaluation and VQA for training).

Author response:
* We provide results in **Table B**. Specifically, we remove QA dialogues from the instruction dataset (including TextQA and common sense QA generated on COCO, reducing about 100k data in total) and test its downstream QA capability on ScienceQA datasets.

  * From Table C, we find both the baseline model and MoE have significant performance drops in the ScienceQA evaluation, indicating MoE can not generalize well to unseen ScienceQA tasks during pre-training. Although generalizing to unseen tasks is not our primary focus, it is an interesting topic that we will research in the future. 

  * Considering the reduction in data amount (from 293k to 193k), there is a performance decrease in most downstream tasks. However, an interesting observation is that in the absence of MoE, the baseline, which is fine-tuned on the instruction dataset without QA datasets, achieves better results on detection tasks, proving the conflict between detection and QA tasks once again.

Reviewer comment (Experiments):
It is unclear how the method would work if it had not seen any QA tasks during training, since it is difficult to understand how a specialized architecture like MoE can generalize well to totally unseen tasks during pre-training.

Author response:
* We provide results in **Table B**. Specifically, we remove QA dialogues from the instruction dataset (including TextQA and common sense QA generated on COCO, reducing about 100k data in total) and test its downstream QA capability on ScienceQA datasets.

  * From Table C, we find both the baseline model and MoE have significant performance drops in the ScienceQA evaluation, indicating MoE can not generalize well to unseen ScienceQA tasks during pre-training. Although generalizing to unseen tasks is not our primary focus, it is an interesting topic that we will research in the future. 

  * Considering the reduction in data amount (from 293k to 193k), there is a performance decrease in most downstream tasks. However, an interesting observation is that in the absence of MoE, the baseline, which is fine-tuned on the instruction dataset without QA datasets, achieves better results on detection tasks, proving the conflict between detection and QA tasks once again.

Reviewer comment (Experiments):
If the model is still trained with VQA but the prompt template of ScienceQA is changed during testing, it is unclear whether the MoE router would be robust to this kind of OoD generalization.

Author response:
* For OOD generalization in QA tasks, we provide experimental results in **Table C**. The query prompt of ScienceQA datasets contains three parts: question, context, and image. We enriched the question and context using GPT-3.5-turbo, respectively. We evaluate the ScienceQA Acc@1 using the 2D Octavius model.

  **Table C**

  | type | Default wo/ enrichment | Enriched contexts | Enriched questions | Enriched contexts + questions |
  | --- | --- | --- | --- | --- |
  | results | 46.95 | 47.58  | 47.43  | 47.03  |

  * Our model shows improvements in both enriched contexts and questions, demonstrating strong OOD generalization and robustness of LoRA-MoE on varying questions and contexts. We also pick some examples [here](    https://github.com/iclrxoct/iclr2024xoct/blob/main/3GST_2.png).

  * We also provide the comparison of the routing weights between the default template and the enriched template [here](      https://github.com/iclrxoct/iclr2024xoct/blob/main/3GST_3.PNG). **It is evident that regardless of the enriched parts, our model consistently selects similar gates with similar weights.** This observation serves as evidence for the robustness of our model in addressing VQA questions/contexts with different patterns.

* We will add the tables, illustrations, and analyses to the final version.

Reviewer comment (Presentation):
In Fig. 5, it is unclear whether a single layer is selected for visualization or an average of all layers is taken.

Author response:
Octavius applies a global sample-based gating network, i.e., we employ one gating network to route LoRA experts in all layers, and Figure 5 illustrates overall gating scores.

Reviewer comment (Reproducibility):
In Tab. 6, the authors claim that load balancing loss has a negative effect. It is unclear whether load balancing is used afterwards, and if not, how to prevent the MoE architecture from collapsing to always using a single expert.

Author response:
* We do not use a load balancing strategy afterwards. As mentioned in the section 4.3 (ablation), sample-based gate is incompatible with the load balancing strategy. For example, it is more reasonable to assign detection samples to a LoRA expert proficient in localizing than the other experts for the purpose of load balancing. Unless the dataset itself is balanced, it's difficult to equally assign experts. Additionally, the experimental results in Table 6 confirm this viewpoint. The load balancing strategy seems more suitable for token-based gates in conventional MoE models, but in this tug-of-war scenario, its interpretability is worse. Moreover, the experimental results of using token-based gates (also with load balancing strategy) are poor in Table 6.

* We claim the load balancing issue as a limitation in the paper and will attempt to address it in future work.

Reviewer comment (Experiments):
The 2D results in Tab. 4 have a significant performance gap with state-of-the-art VLLMs like InstructBLIP (e.g., ScienceQA and Flickr30K captioning).

Author response:
InstructBLIP is built on powerful BLIP-2 (Vicuna-13b, caption 71.6 & ScienceQA 61.0, training with 129M caption/QA datasets), so it is reasonable that InstructBLIP has better results in caption and ScienceQA tasks.

Reviewer comment (Experiments):
The authors should have utilized stronger baseline methods, as there is a significant performance gap between LAMN and state-of-the-art methods including InstructBLIP and Shikra.

Author response:
* InstructBLIP

  * Compared with LLaVA/LAMM, InstructBLIP employs a more powerful 2D visual encoder (CLIP + BLIP-2 pretrained Q-former) while keeping the entire LLM frozen. Therefore, InstructBLIP and our LoRA-MoE are compatible and the LoRA/LoRA-MoE implementation of InstructionBLIP is feasible. For example, InstructBLIP freezes its LLM during instruction tuning, so we can add a third stage to fine-tune LLM with LoRA/LoRA-MoE while keeping other components frozen.

  * However, although InstructBLIP mentions their templates to construct instruction data in the paper, a complete instruction-following dataset meta file has not been released. We cannot guarantee reproducing the results of InstructBLIP. You can find many issues in the InstructBLIP repository related to replication, e.g. [#302](https://github.com/salesforce/LAVIS/issues/302), [#439](https://github.com/salesforce/LAVIS/issues/439), [#542](https://github.com/salesforce/LAVIS/issues/542), and more. We will continuously track the feedback on the InstructBLIP Github repository, reproduce the results from its paper, and implement a LoRA/LoRA-MoE based InstructBLIP in the future.

* Shikra

  * Shikra applies a full fine-tuning scheme, we cannot guarantee the LoRA implementation of Shikra is effective.

  * Besides, Shikra mainly focuses on 2D grounding tasks. Although Shikra is capable of performing other tasks such as captioning and VQA, it does not conduct a zero-shot evaluation on captioning/VQA tasks (test datasets, Flicker30k, and VQAv2, are included in the training set). If we directly remove related datasets (Flickr30k, VQAv2) from Shikra's training set for zero-shot evaluation, we cannot guarantee the zero-shot performance of Shikra on captioning/VQA tasks.

* In addition to the mentioned InstructBLIP and Shikra, KOSMOS series [1-2] also faces replication difficulties (complete datasets + models not released + too excessive computational resources). Other 2D MLLM methods do not significantly outperform LLaVA and LAMM. Considering the simplicity of LLaVA and LAMM, and both of them implementing LoRA-based MLLM, we choose these two methods as our baseline. **Furthermore, we will continue to follow the developments in 2D MLLMs, and if a better baseline model emerges with sufficient transparency to support our work, we will adopt it.**

[1] Language Is Not All You Need: Aligning Perception with Language Models. 2023, Huang et al.

[2] KOSMOS-2: Grounding Multimodal Large Language Models to the World. 2023, Peng et al.

Reviewer comment (Experiments):
I am primarily concerned by the analysis in Figure 5 -- it seems that all the 2D tasks are using only two experts! This makes me skeptical about the utility of MoE at all. Could you run the ablation in Table 5 with 2 experts? So these two experts will get selected each time and there is routing involved, but the model has the capacity to learn with 2 LoRAs at once instead of one (which is what it seems to be doing in Fig5).

Author response:
* We use top-2 gate for routing, i.e., selecting two experts from all experts for predicting. It is a very popular and common gating strategy used in MoE models [1-4]. 

* We provide the results of the model with 2 experts in **Table A**. In the "2-learnable" experiment, we apply normalized gating scores (a 2-dim tensor, e.g., [0.34, 0.66]) as expert weights, while in "2-average" experiments, we use [0.5, 0.5] as expert weights.

  **Table A**

  | Gate Type| #Experts | LoRA-Rank  | VOC Recall@ 0.5  | VOC Prec@ 0.5  | ScienceQA Acc@1 | # Trainable Params. |
  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
  | Baseline | 1 | 32 | 7.61 | 5.95 | 40.31 | 0.4% |
  | top-2 | 4 | 32 | 39.04 | 35.21 | 46.95 | 1.6% |
  | top-2 | 2-learnable | 32 | 15.03 | 8.63 | 41.40 | 0.8% |
  | top-2 | 2-average | 32 | 7.28 | 5.32 | 40.80 | 0.8% |

  * We can observe a huge performance gap with 2 experts in detection (VOC) tasks. For the "2-average" experiment that directly applies [0.5, 0.5] as expert weights, it can be seen as simply increasing the model parameters. For example, you can replace 2 experts with a larger LoRA', i.e., LoRA' = 0.5*LoRA1 + 0.5*LoRA2. For the "2-learnable" experiment that applies weights generated by the gate, the gate can learn more kinds of combinations of routing weights for different tasks. To some extent, different combinations of routing weights can be seen as different routing results. Therefore, it achieves better results than the "2-average" experiment. 

  * When we use more experts (from 2 to 4), the gate has more choices and achieves the best results in **Table A**. However, as more experts are introduced, as mentioned in the section 4.3 (ablation), the model will be affected by the load balancing issues, leading to a decline in the utilization rate of different experts (this is why LoRA#2 is not used in Figure 5). Allocating too many tasks to the same experts eventually leads to a performance drop (8 experts perform lower than 4 experts in Table 5).

* We will add the above results in Table 5 in the final version.

[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. 2017, Shazeer et al.

[2] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. 2022, Lepikhin et al.

[3] GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. 2022, Du et al.

[4] Lifelong Language Pretraining with Distribution-Specialized Experts. 2023, Chen et al.

Reviewer comment (Experiments):
An additional analysis that is needed is how the gate routing is distributed between 2D and 3D tasks, for the model that is trained on the combined LAMMv2+ScanNet instruction tuning dataset.

Author response:
* We provide the visualization of routing results on the combined LAMMv2 + Scan2Inst [here](    https://github.com/iclrxoct/iclr2024xoct/blob/main/6dGg_3.png).

* In the 2D+3D setup, there are still load balancing issues (#2 and #5) in the routing distribution like the 2D setup. (see the discussion of the load balancing issues in section 4.3, ablation)

* In the 3D tasks, 3D captioning and 3D classification mainly focus on instance-level tasks, such as the caption or category of a specific object (see Figure 1 in the main part of the paper and Figure 4 in the appendix), while 3D VQA focuses more on relationships between multiple objects in the scene and the understanding of the entire scene. Therefore, there are two patterns of routing weights between 3D Cap./3D Cls and 3D VQA.

* Some experts engage in knowledge sharing across different modalities (#2), while others may be more inclined towards a single modality (#1 and #5).

Reviewer comment (Evaluation):
Given the result in Figure 5, I am not convinced about the utility of MoE with routing, when it doesn't seem that different experts are even used.

Author response:
* First, non-MoE baselines in Table 2-3 train additional LoRAs from scratch (one LoRA per layer) while keeping the pre-trained weights of Vicuna-13b frozen following LAMM [1]. This is a common PEFT training setup for LLMs.

* In Figure 5, Classification/Caption/FacialAttribute share similar expert selection but have different weights. It is because, **although these tasks have different downstream applications, they are very similar in instructions and responses.** I can give some examples below:

  * Captioning:

    **Q**: What is the caption of the image? 

    **A**: The image depicts an older man and a young man in a close embrace. The older man, who is bald and wearing a red shirt, appears to be hugging the younger man. They are standing in front of a black wall, which enhances the contrast of their positions.

  * Classification:

    **Q**: What is the most accurate category label for this image?
    
    **A**: The image depicts a cute and adorable small white fluffy dog.

  * Facial Attribute

    **Q**: What color is the person's hair in the image?

    **A**: The image shows a young woman with dark, long, and curly hair.

  From the above examples, we can find that in classification/facial attribute tasks, **LLMs are inclined to provide detailed descriptions of the target object (including color, shape, adjectives, etc.) rather than just basic categories**. The responses to these tasks are very similar to some simple captioning tasks, so it is reasonable that the gating network makes similar expert selections for captioning/classification/facial attribute tasks. Instead, in very different tasks like detection, gating networks generate different expert selections.

[1] Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. 2023, Yin et al.

Reviewer comment (Experiments):
I am not sure what the non-MoE baseline in Tables 2-3 is -- is it merely training the frozen model minus the LoRA parameters?

Author response:
* First, non-MoE baselines in Table 2-3 train additional LoRAs from scratch (one LoRA per layer) while keeping the pre-trained weights of Vicuna-13b frozen following LAMM [1]. This is a common PEFT training setup for LLMs.

* In Figure 5, Classification/Caption/FacialAttribute share similar expert selection but have different weights. It is because, **although these tasks have different downstream applications, they are very similar in instructions and responses.** I can give some examples below:

  * Captioning:

    **Q**: What is the caption of the image? 

    **A**: The image depicts an older man and a young man in a close embrace. The older man, who is bald and wearing a red shirt, appears to be hugging the younger man. They are standing in front of a black wall, which enhances the contrast of their positions.

  * Classification:

    **Q**: What is the most accurate category label for this image?
    
    **A**: The image depicts a cute and adorable small white fluffy dog.

  * Facial Attribute

    **Q**: What color is the person's hair in the image?

    **A**: The image shows a young woman with dark, long, and curly hair.

  From the above examples, we can find that in classification/facial attribute tasks, **LLMs are inclined to provide detailed descriptions of the target object (including color, shape, adjectives, etc.) rather than just basic categories**. The responses to these tasks are very similar to some simple captioning tasks, so it is reasonable that the gating network makes similar expert selections for captioning/classification/facial attribute tasks. Instead, in very different tasks like detection, gating networks generate different expert selections.

[1] Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. 2023, Yin et al.

Reviewer comment (Experiments):
There is no analysis on how the routing among experts actually works. It would be great if the authors can provide some qualitative study of the predictions from sample-based gating network as responses to the input task, to show how the routing mechanism work. I wonder whether the gating network will simply act like a task classifier, or it's not the case.

Author response:
* We provide the visualization of routing weights on 2D setup in Figure 5.

  * In most downstream tasks, the questions of each task are the same in evaluation. Therefore, Figure 5 can be seen as the predictions from the gating network as responses to the input tasks. The only exception is ScienceQA task where different questions lead to different routing weights. We analyze the generated weights for all questions in ScienceQA and find that 3 experts (#0/1/3) are chosen in ScienceQA, but most of them are composed of LoRA#1 and LoRA#3. Therefore, we use the normalized average weights of #1 and #3 for visualization.

  * Furthermore, we find that captioning/classification/facial attribute tasks share similar expert selection in Figure 5. It is because, although these tasks have different downstream applications, they are very similar in instructions and responses. I can give some examples below:

    * Captioning:

      **Q**: What is the caption of the image? 

      **A**: The image depicts an older man and a young man in a close embrace. The older man, who is bald and wearing a red shirt, appears to be hugging the younger man. They are standing in front of a black wall, which enhances the contrast of their positions.

    * Classification:

      **Q**: What is the most accurate category label for this image?
    
      **A**: The image depicts a cute and adorable small white fluffy dog.

    * Facial Attribute

      **Q**: What color is the person's hair in the image?

      **A**: The image shows a young woman with dark, long, and curly hair.

    From the above examples, we can find that in classification/facial attribute tasks, LLMs are inclined to provide detailed descriptions of the target object (including color, shape, adjectives, etc.) rather than just basic categories. They are similar to some simple captioning tasks, so it is reasonable that the gating network makes similar expert selections for captioning/classification/facial attribute tasks. **In these cases, the gating network can learn to allocate similar experts to similar but different tasks, and it can not be simply regarded as a task classifier. While in very different tasks like detection, the gating network tends to generate different expert selections.**

* We also provide additional visualization on 2D+3D setup [here](https://github.com/iclrxoct/iclr2024xoct/blob/main/6dGg_3.png).

  * In the 2D+3D setup, there are still load balancing issues (#2 and #5) in the routing distribution like the 2D setup. (see the discussion of the load balancing issues in section 4.3, ablation)

  * In the 3D tasks, 3D captioning and 3D classification mainly focus on instance-level tasks, such as the caption or category of a specific object (see Figure 1 in the main part of the paper and Figure 4 in the appendix), while 3D VQA focuses more on relationships between multiple objects in the scene and the understanding of the entire scene. Therefore, there are two patterns of routing weights between 3D Cap./3D Cls and 3D VQA.

  * Some experts engage in knowledge sharing across different modalities (#2), while others may be more inclined towards a single modality (#1 and #5).

Reviewer comment (Experiments):
The scaling behavior as more modalities and tasks are added is not studied. There may be limitations in very high multi-task settings.

Author response:
* In the 2D/3D/2D+3D setups, our work already covers a wide range of tasks in most 2D/3D scenarios, including captioning, VQA, classification, etc., in both instruction tuning and downstream evaluations. Recent works such as VL-BERT[1], QWen-VL[2], and VisionLLM[3] involve 2D visual tasks that are similar to ours. If we want to further expand more datasets in these setups, it would primarily be expansions in terms of data amount.  Since this is a very high multitask learning in these setups, we believe that we can adequately demonstrate the generalization capability of LoRA-MoE in multitask learning and multimodal learning on the existing datasets and modalities. 

* Considering the workload, we will further explore more modalities on this topic beyond 3D like audio, and robotic tasks in future works.

[1] Vl-beit: Generative vision-language pretraining. 2022, Bao et al.

[2] Qwen-vl: A frontier large vision-language model with versatile abilities. 2023, Bai et al.

[3] Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Wang et al.

Reviewer comment (Evaluation):
Can you show some qualitative study of the predictions from the sample-based gating network as responses to the input task? It can help us understand how network routing works.

Author response:
* We provide the visualization of routing weights on 2D setup in Figure 5.

  * In most downstream tasks, the questions of each task are the same in evaluation. Therefore, Figure 5 can be seen as the predictions from the gating network as responses to the input tasks. The only exception is ScienceQA task where different questions lead to different routing weights. We analyze the generated weights for all questions in ScienceQA and find that 3 experts (#0/1/3) are chosen in ScienceQA, but most of them are composed of LoRA#1 and LoRA#3. Therefore, we use the normalized average weights of #1 and #3 for visualization.

  * Furthermore, we find that captioning/classification/facial attribute tasks share similar expert selection in Figure 5. It is because, although these tasks have different downstream applications, they are very similar in instructions and responses. I can give some examples below:

    * Captioning:

      **Q**: What is the caption of the image? 

      **A**: The image depicts an older man and a young man in a close embrace. The older man, who is bald and wearing a red shirt, appears to be hugging the younger man. They are standing in front of a black wall, which enhances the contrast of their positions.

    * Classification:

      **Q**: What is the most accurate category label for this image?
    
      **A**: The image depicts a cute and adorable small white fluffy dog.

    * Facial Attribute

      **Q**: What color is the person's hair in the image?

      **A**: The image shows a young woman with dark, long, and curly hair.

    From the above examples, we can find that in classification/facial attribute tasks, LLMs are inclined to provide detailed descriptions of the target object (including color, shape, adjectives, etc.) rather than just basic categories. They are similar to some simple captioning tasks, so it is reasonable that the gating network makes similar expert selections for captioning/classification/facial attribute tasks. **In these cases, the gating network can learn to allocate similar experts to similar but different tasks, and it can not be simply regarded as a task classifier. While in very different tasks like detection, the gating network tends to generate different expert selections.**

* We also provide additional visualization on 2D+3D setup [here](https://github.com/iclrxoct/iclr2024xoct/blob/main/6dGg_3.png).

  * In the 2D+3D setup, there are still load balancing issues (#2 and #5) in the routing distribution like the 2D setup. (see the discussion of the load balancing issues in section 4.3, ablation)

  * In the 3D tasks, 3D captioning and 3D classification mainly focus on instance-level tasks, such as the caption or category of a specific object (see Figure 1 in the main part of the paper and Figure 4 in the appendix), while 3D VQA focuses more on relationships between multiple objects in the scene and the understanding of the entire scene. Therefore, there are two patterns of routing weights between 3D Cap./3D Cls and 3D VQA.

  * Some experts engage in knowledge sharing across different modalities (#2), while others may be more inclined towards a single modality (#1 and #5).

Reviewer comment (Experiments):
Given our awareness of each example's task, an important baseline involves employing a dedicated LoRA for each task individually.

Author response:
**Table A**
|method|VOC Recall@ 0.5|ScienceQA Acc@1|Flickr30k CIDEr|CIFAR-10 Acc@1|FacialAttr Smile Acc@1|
|:----:|:----:|:----:|:----:|:----:|:----:|
|LAMM (LoRA)|7.61|40.31|0.21|73.50|50.15|
|LAMM (LoRA-MoE)|39.04|46.95|5.66|65.40|59.82|
|LAMM (LoRA-individual)|28.38|48.54|5.83|82.11|*28.76|

* We provide results in **Table A**. Specifically, we classify the 2D visual tasks as 6 types, i.e., QA, description, classification, detection, n-round conversations, and others, in instruction datasets, and employ 6 LoRAs for each task individually for joint training. During evaluation, we manually assign the expert corresponding to the most similar tasks in training to the downstream tasks.
 
* The problem with employing separate LoRA for each task is that **it is difficult to assign a suitable expert for downstream tasks that are not present in instruction datasets**, e.g., Facial Attribute Recognition. In this task, we attempted to assign "others" or "description" experts to it, but both results were not good (see * in the table, we selected the "others" expert because it is relatively better). Moreover, **manually inputting the task type or the selected expert into the model during inference is not as efficient as the automatic gating based on questions in LoRA-MoE**.

* Besides, **if we have no idea what downstream tasks will be performed, it is challenging to design a LoRA for each training task due to potential generalization issues**.

Reviewer comment (Experiments):
Conducting an ablation study on the impact of top-k would be informative.

Author response:
**Table B**
|Gate|#Experts|VOC Recall@ 0.5|VOC Prec@ 0.5|ScienceQA Acc@1|
|:----:|:----:|:----:|:----:|:----:|
|-|1|7.61|5.95|40.31|
|top-1|4|22.42|21.23|36.88|
|top-2|4|39.04|35.21|46.95|
|top-3|4|38.57|36.02|43.89|

* We provide results of adopting different top-k gates in **Table B**. We will add this table in the final version.

* From the table, we can find:
  * Top-1 gate has a poor performance due to its less flexibility in choice (4 combinations) compared with top-2/top-3 gate. 
  * While in the top-3 gate scenario, we find that the third choice is relatively small in most tasks (e.g., in classification tasks, the routing weight is [0.1, 0, 0.4, 0.5]), therefore it is very similar to the top-2 gate, and achieves similar results as the top-2 gate.

Reviewer comment (Theory):
How can you attain top-2 sparsity gates while ensuring compatibility with gradient flow?

Author response:
* We implement the top-2 gate based on an open-source implementation of [1] from [repo1](https://github.com/lucidrains/mixture-of-experts/tree/master) and [repo2](https://github.com/davidmrau/mixture-of-experts). We can give you a simple PyTorch-like pseudo code for top-2 gate implementation.
  
  ```python
  class Top2Gate(nn.Module):
      def __init__(self, num_experts, ...):
          self.num_experts = num_experts
          self.w_gating = nn.Parameter(torch.randn(input_dim, num_experts))
          ...
        
      @staticmethod
      def top1(tensor):
          return tensor.topk(k=1, dim=-1)
        
      def forward(self, x):  # x: bs, n_token, dim
          gate = torch.einsum('bnd,de->bne', x, self.w_gating)  # gate: bs, n_token, n_experts
          gate = gate.mean(dim=-1).softmax(dim=-1)  # reduce token, gate: bs, n_experts
        
          gate_1, index_1 = self.top1(gate)
          mask_1 = F.one_hot(index_1, self.num_experts)
        
          gate_without_top1 = gate * (1. - mask_1)
          gate_2, index_2 = self.top1(gate_without_top1)

          norm = gate_1 + gate_2 + eps
          gate_1 /= norm
          gate_2 /= norm
        
          top2_gate = torch.zeros(x.shape[0], self.num_experts)  # top2_gate: bs, n_experts
          top2_gate.scatter_(1, index_1.view(-1, 1), gate_1.view(-1, 1))
          top2_gate.scatter_(1, index_2.view(-1, 1), gate_2.view(-1, 1))
          return top2_gate
  ```

* Other implementation details:
  
  * Given the input question embedding and N LoRA experts, we can obtain N-dim routing weights using the top-2 gate, where two experts with the highest scores are retained and the others are set to zero. Next, we can calculate the weighted sum of N LoRA experts. The non-activated experts are multiplied by a coefficient of 0 in routing weights.

  * This approach has almost no significant impact on the computation cost when N=4/6 in our setting during fine-tuning. During inference, non-activated experts can be ignored, thus improving efficiency. Furthermore, if the tasks and questions are specified, LoRA-MoE can merge parameter weights with LLM like vanilla LoRA to reduce extra inference costs.

[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. 2017, Shazeer et al.