Paper: Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning (ICLR-2024)

Reviewer comment (Experiments):
**Explicit vs Implicit separation of task-specific and shared modules**: To my understanding, the proposed method can be seen as subcase of some of the baselines mentioned: If the number of adapters in `Poly` is set to $A + BT$, then in theory the model could learn to make certain modules task-specific by setting $w_i^t$ to 1 for only certain pairs of $(i, t)$. In contrast, the proposed method makes this structure explicit by defining task-specific modules; however, this still requires additional $BT$ task-specific adapter modules, hence an increased number of parameters. From the experiments, it is not clear to me if the number of parameters/modules used by the baselines and the proposed method are comparable.

Author response:
We understand the importance of ensuring a fair comparison with baselines, particularly regarding our use of A+B modules versus the A task-shared modules in baseline methods. Initially, our goal was to highlight the significance of adding task-specific modules. Furthermore, we presented an analysis of the number of trainable parameters in Figure 3 to elucidate this aspect. To foster a more equitable comparison, as mentioned in our general comment, we modified our experimental setup to (A−B + B) modules. This adjustment aligns the number of modules with baseline configurations, ensuring a more appropriate and rigorous comparison.

Reviewer comment (Experiments):
The number of task-specific modules $B$ for each task seems to be an important parameter, but I did not see much discussion or ablation about it; it seems that $B$ is set to 1 throughout the paper.

Author response:
Our initial choice of B=1 for task-specific modules was made to streamline the model's configuration and facilitate analysis. However, in response to your suggestion, we revisited this aspect in Ablation Study 1. We explored and demonstrated the performance variations with different allocations of task-specific modules, providing insights into the impact of varying B on our model's effectiveness.

Reviewer comment (Experiments):
The proposed method can use $A + B$ modules per task, while baselines only consider $A$ task-shared modules. It is not clear to me if the setup is fair to the baselines. Considering other configurations would be interesting.

Author response:
We understand the importance of ensuring a fair comparison with baselines, particularly regarding our use of A+B modules versus the A task-shared modules in baseline methods. Initially, our goal was to highlight the significance of adding task-specific modules. Furthermore, we presented an analysis of the number of trainable parameters in Figure 3 to elucidate this aspect. To foster a more equitable comparison, as mentioned in our general comment, we modified our experimental setup to (A−B + B) modules. This adjustment aligns the number of modules with baseline configurations, ensuring a more appropriate and rigorous comparison.

Reviewer comment (Evaluation):
From the results in **Table 2**, it seems that the performance improvement is not consistent: for instance, with the encoder-decoder `T5` model, improvement mainly comes from the `WiC` task, which could mean this task is more likely to interfere with others. However, for the decoder-only `GLM`, we see that adding a task-specific module yields significant improvement across all tasks. It is not clear to me why that is the case: since the tasks are the same in both cases, it does not seem to be directly related to the multi-task setup; could it be related to the number of LoRA modules with respect to the architecture (*see point 2*)?

Author response:
Thank you for your professional insight. We have also noticed this point. Considering the significant difference in parameter size between FLAN-T5-Large and GLM-10B, we believe that when the parameter size is large, the ability to learn representations is stronger, which can support better fitting of more tasks. 

However, the parameter size of FLAN-T5-Large is insufficient to support effective optimization of all tasks. In the Appendix, we have added comparative experiments on different PEFT methods on FLAN-T5-XL on SuperGLUE. 

The experimental results have confirmed that after expanding the parameter size, the model's ability to fit multiple tasks becomes stronger, and our method effectively reduces negative transfer phenomena, achieving robust improvements in average and subtasks.

The appendix table is shown as: 
|             | **avg** | **boolq** | **cb** | **copa** | **multirc** | **rte** | **wic** | **wsc.fixed** |
|-------------|---------|-----------|--------|----------|-------------|---------|---------|---------------|
| lora        | 82.10   | 89.57     | 78.57  | 96.00    | 79.62       | 87.36   | 67.65   | 75.96         |
| moe-lora    | 82.97   | 89.32     | 78.57  | 96.00    | 83.76       | 89.16   | 68.02   | 75.96         |
| poly-lora   | 84.29   | 89.51     | 80.35  | 96.00    | 84.63       | 90.97   | 68.80   | 79.80         |
| mhr-lora    | 84.39   | 89.51     | 80.35  | 97.00    | 84.61       | 90.97   | 68.49   | 79.80         |
| cpoly-lora  | **84.94**   | 89.55     | **82.14**  | **97.00**    | **84.96**       | **91.69**   | 68.50   | **80.76**         |

Reviewer comment (Experiments):
**Note post-rebuttal:** I still have some reserves about the method's robustness and sensitivity to the task specific/shared setup ratio, but in light of the rebuttal addressing my main concerns and added ablation experiments, I'll raise my score from 5 to 6.

Author response:
Dear Reviewer,

Thank you once again for your valuable time and insights. We are pleased that our additional ablation studies have effectively addressed your main concerns.

We acknowledge and appreciate your reservations regarding the robustness of our method and the tuning of the $(A, B)$ configuration settings. We intend to explore these aspects more thoroughly in our ongoing and future research endeavors, ensuring a comprehensive examination and refinement.

Thank you again for your constructive feedback, which has been instrumental in enhancing our work.

Sincerely,

Authors

Reviewer comment (Writing):
I found the introduction hard to read as it mixes introduction with some related work towards the end, and introduces several concepts without really contextualizing them (e.g., in two paragraphs, the text jumps from MoLora $\rightarrow$ Poly $\rightarrow$ MHR $\rightarrow$ CGC and PLE).

Author response:
**Clarification on Concept Introduction:**

Your observation about the introduction section is well-received. Our intention was to draw parallels between existing PEFT methods (MoLoRA, Poly, MHR) and MoE concepts (MoE, MMoE, CGC, PLE) to provide a quick overview of how our method (CPoly) aligns with and differs from these approaches. We will ensure that each concept is introduced more gradually and is clearly contextualized.

Reviewer comment (Writing):
Using $A$ and $B$ in Equation 5 is a bit confusing as these were introduced earlier as the number of shared and task-specific modules.

Author response:
**Clarification on Equation Notation:**

Regarding your point about the notation in Equation 5, we acknowledge the confusion caused by using $A$ and $B$, and have changed it to $W_{down}$ and $W_{up}$ for clarification.

Reviewer comment (Evaluation):
The method does not achieve significant improvement with T5 architecture. (But I am not expert in NLP tasks and I am open to other reviewers' opinions regarding the performance.)

Author response:
Thank you for your observations regarding the performance improvements with the T5 architecture and the distinction between the performance with GLM-10B and T5 models.

To discern the impact of architecture versus model capacity, we conducted **Ablation Study 3**, which included a range of T5 and FLAN-T5 models with varying capacities (Large/0.78B , XL/3B, XXL/11B) - the XXL version is comparable with GLM-10B. The results demonstrated that our method consistently outperforms other methods across these scales. For instance, with T5-XXL, our method achieved significant gains in performance, suggesting that while model capacity plays a role, the architecture of our method effectively utilizes this additional capacity. On the other hand, the performance saturation observed, especially with FLAN-T5, might partly stem from an overlap between its training dataset and the SuperNI tasks.

Reviewer comment (Novelty):
The model design seems incremental to the previous methods Poly and MHR.

Author response:
Thank you for your comment regarding the comparison of our model design with previous methods like Poly and MHR. While our approach builds upon the foundations laid by these methods, it also introduces significant advancements that distinguish it from its predecessors.

A key innovation in our method is the strategic integration of task-specific skills alongside task-common skills. This dual-skill framework is not just an extension but a significant departure from methods like Poly and MHR, which predominantly focus on shared skills across tasks. Our approach enables a more nuanced and targeted application of skills to specific tasks, enhancing overall performance and adaptability.  

Our ablation studies demonstrate the efficacy of this approach. For example, **Ablation Study 1** showed notable improvements in performance with the optimal balance of task-common and task-specific skills, highlighting the effectiveness of our model's unique configuration.

In summary, while our model design is inspired by earlier methods, it offers a more flexible and scalable solution for complex multi-task environments.

Reviewer comment (Reproducibility):
It is unclear about the optimization loss function and the main paper does not discuss this.

Author response:
Thank you for bringing to our attention the lack of the optimization loss function in our manuscript. We apologize for this oversight. In our method, we have employed the cross-entropy loss function, a standard and widely used choice for language models. The language model predicts next tokens autoregressively. It's essentially a multi classification problem, where the classes are the choices of tokens in the sequence.

Reviewer comment (Evaluation):
Since you have used Gumbel Sigmoid to optimize the w_i, what is the distribution of all w_i's learnt in the model? Is there any specific loss to force w_i to be close to 0 or 1? What is the performance variance if you need sample the w_i based on Bernouli distributions in the final evaluation?

Author response:
During training, we utilize the Gumbel Sigmoid, or Relaxed Bernoulli in binary cases, to smooth the logits derived from w_i. This method enables differentiable sampling from a Bernoulli distribution, crucial for gradient-based optimization in neural networks. In the evaluation phase, we apply a Sigmoid function for smoothing. It's noteworthy that we didn't implement a specific loss function to directly influence this sampling phase. Recognizing the potential impact of such a loss function, we see this as a valuable direction for future research.

Reviewer comment (Experiments):
In the experiment results, it is clear that the method with GLM-10B outperforms baselines a lot and the method with T5 stands close to the baselines. In the paper, the authors claim the difference is due to the different model architecture. However, GLM-10B and T5_large also differ a lot in the model capacity. How do you know the difference of performance compared to the baselines is due to the architecture instead of model capacity?

Author response:
Thank you for your observations regarding the performance improvements with the T5 architecture and the distinction between the performance with GLM-10B and T5 models.

To discern the impact of architecture versus model capacity, we conducted **Ablation Study 3**, which included a range of T5 and FLAN-T5 models with varying capacities (Large/0.78B , XL/3B, XXL/11B) - the XXL version is comparable with GLM-10B. The results demonstrated that our method consistently outperforms other methods across these scales. For instance, with T5-XXL, our method achieved significant gains in performance, suggesting that while model capacity plays a role, the architecture of our method effectively utilizes this additional capacity. On the other hand, the performance saturation observed, especially with FLAN-T5, might partly stem from an overlap between its training dataset and the SuperNI tasks.

Reviewer comment (Theory):
As the adapter in LORA are low-rank linear projection of parameters in attention modules, a combination of task-common and task-specific LORA adapters seems to be equivalent to just Mixture of LORA as addition of linear transformations is still a linear transformation (rows 2 and 3). Thus, it is unclear why decomposing learnable parameters would improve performance.

Author response:
Thank you for your insightful observation about the linear nature of these transformations. However, the key advantage of our approach lies in the strategic weighting and targeted application of these adapters.

1. **Task-Specific Tailoring**: The introduction of task-specific adapters enables our model to customize its responses to the unique requirements of individual tasks, a level of specificity and nuance that general, task-agnostic adapters may not achieve. This allows the model to effectively address task-specific features and variations.

2. **Dynamic Updating of Components**: The task-common adapters are updated based on the entire multi-task dataset, providing a broad base of learning, while each task-specific adapter is fine-tuned using only data from its respective individual task. This dual approach ensures both generalization and specialization in the model's learning process.

3. **Evidence from Experiments**: Our experimental results demonstrate that this method of decomposing and dynamically recombining parameters leads to measurable performance improvements.

In conclusion, our model's performance gains stem from the thoughtful combination of task-common and task-specific adapters. This approach not only leverages the strengths of each adapter type but also ensures a more nuanced and effective adaptation to a diverse array of tasks.

Reviewer comment (Experiments):
The experimental section misses some studies to show the effective of hyper-parameters in the model. For examples, what is the how number of adapters in task-common and task-specific modules effect the performances? What is the impact of rank or the number of tasks toward final performance? These experiments would offer better insight into how robust the proposed method is under different setting.

Author response:
Thank you for pointing out the necessity of further experimental insights into the effectiveness of various hyper-parameters in our model. We agree that understanding the impact of the number of adapters, tasks, and rank is crucial for assessing the robustness and limitations of our approach. In response to your concerns, we refer to **Ablation Studies 1 and 2**, which delve into these aspects:

**Ablation Study 1**: This study explored how the number of adapters in task-common and task-specific modules affects performance. For instance, in our experiments with all 3 models, we observed a notable performance increase when using 3 common and 1 task-specific skill compared to 4 common skills. However, further increasing the number of task-specific skills showed diminishing returns.

**Ablation Study 2**: Our method consistently outperformed others across these varying task numbers, demonstrating robust scalability.

Regarding the impact of rank on our model's performance, we based our experimental setup on insights from the [original LoRA paper](https://arxiv.org/abs/2106.09685). In selecting LoRA as our adapter unit, we were guided by the findings in their study, which indicated that variations in rank might not have a significant impact on model performance for each individual LoRA.

Reviewer comment (Experiments):
The performance seems to saturate when being applied to strong base model such as T5-Large. It seems to suggest that the effect of task-common components vanish when the base model can generalize well toward different downstream tasks. This could be an interesting phenomena can be study using stronger base model such as llama and llama2.

Author response:
As mentioned in our general comment, the base model used in our original manuscript was FLAN-T5-Large. The performance saturation noted may be partly due to the overlap between FLAN-T5's training dataset and the tasks in the SuperNI dataset. To mitigate this and improve comparability, we expanded our ablation studies, particularly **Ablation Study 3**, to encompass a broader range of models including T5-Large (0.78B), T5-XL (3B), and stronger one T5-XXL (11B), and their FLAN-T5 counterparts. These studies demonstrate that as the model scale increases, our method continues to outperform others, indicating its effectiveness across different model capacities.

Regarding the suggestion to use models like Llama and Llama2 for stress-testing our framework, we find it intriguing. Our choice to focus on T5 and GLM was to enable direct comparison with existing results in the manuscript. Exploring our approach with advanced model is an important direction for our future research to further understand and enhance our methodology.

Reviewer comment (Reproducibility):
Could the authors clarify on how they combine different adapters? This would help with understanding as well as reproducibility.

Author response:
To clarify, we employ linear combinations for combining the output of different adapters in our model, as detailed in equation (1) of our manuscript. To further aid in understanding and reproducibility, we are planning to open-source our code.

Reviewer comment (Experiments):
Experiments on the effects of the number of adapters/ranks/tasks could provide better insight on the robustness and limitations of the proposed methods.

Author response:
Thank you for pointing out the necessity of further experimental insights into the effectiveness of various hyper-parameters in our model. We agree that understanding the impact of the number of adapters, tasks, and rank is crucial for assessing the robustness and limitations of our approach. In response to your concerns, we refer to **Ablation Studies 1 and 2**, which delve into these aspects:

**Ablation Study 1**: This study explored how the number of adapters in task-common and task-specific modules affects performance. For instance, in our experiments with all 3 models, we observed a notable performance increase when using 3 common and 1 task-specific skill compared to 4 common skills. However, further increasing the number of task-specific skills showed diminishing returns.

**Ablation Study 2**: Our method consistently outperformed others across these varying task numbers, demonstrating robust scalability.

Regarding the impact of rank on our model's performance, we based our experimental setup on insights from the [original LoRA paper](https://arxiv.org/abs/2106.09685). In selecting LoRA as our adapter unit, we were guided by the findings in their study, which indicated that variations in rank might not have a significant impact on model performance for each individual LoRA.

Reviewer comment (Experiments):
Stronger base model can be used to stress-test the generalizability of the proposed framework.

Author response:
As mentioned in our general comment, the base model used in our original manuscript was FLAN-T5-Large. The performance saturation noted may be partly due to the overlap between FLAN-T5's training dataset and the tasks in the SuperNI dataset. To mitigate this and improve comparability, we expanded our ablation studies, particularly **Ablation Study 3**, to encompass a broader range of models including T5-Large (0.78B), T5-XL (3B), and stronger one T5-XXL (11B), and their FLAN-T5 counterparts. These studies demonstrate that as the model scale increases, our method continues to outperform others, indicating its effectiveness across different model capacities.

Regarding the suggestion to use models like Llama and Llama2 for stress-testing our framework, we find it intriguing. Our choice to focus on T5 and GLM was to enable direct comparison with existing results in the manuscript. Exploring our approach with advanced model is an important direction for our future research to further understand and enhance our methodology.