Paper: Mixture of LoRA Experts (ICLR-2024)

Reviewer comment (Experiments):
Could you provide more results on different NLP datasets and tasks?

Author response:
**A1 & A2**. We conducted extensive experiments across various tasks, including **Translation**, **Struct to Text**, **Closed-Book QA**, and multiple subtasks within the challenging **Big-Bench Hard (BBH)** datasets. Additionally, we introduced a new LoRA merging variant called **PEMs** $^{[1]}$, recently proposed in the field of NLP.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of **3.8** over LoRAHub and outperforms PEMs by a substantial margin of **9.0**.

In the domain of natural language generation tasks (**Translation** and **Struct to Text**), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by **1.5** and PEMs by **2.7**. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of **2.1** over LoRAHub and **2.6** over PEMs. These results highlight the effectiveness of our model in generation task.



| Task                            | Metric  | LoRAHub  | PEMs     | Our      |
| ------------------------------- | ------- | -------- | -------- | -------- |
| **Translation**                 |         |          |          |          |
| WMT '14 En->Fr                  | BLEU    | 27.4     | 25.6     | **29.1** |
| WMT '14 Fr->En                  | BLEU    | 29.4     | 27.1     | **31.3** |
| WMT '16 En->De                  | BLEU    | 24.6     | 24.9     | **27.7** |
| WMT '16 De->En                  | BLEU    | **29.9** | 28       | 29.1     |
| WMT '16 En->Ro                  | BLEU    | 17.7     | 15.2     | **18.9** |
| WMT '16 Ro->En                  | BLEU    | 23.5     | 21.7     | **25.1** |
| Average                         | BLEU    | 25.4     | 24.2     | **26.9** |
| **Struct to Text**              |         |          |          |          |
| CommonGen                       | Rouge-1 | 53.7     | 48.8     | **55.1** |
|                                 | Rouge-2 | **23.1** | 22.4     | 23.1     |
|                                 | Rouge-L | 49.7     | 47.2     | **53.9** |
| DART                            | Rouge-1 | 45.3     | 46.2     | **48.8** |
|                                 | Rouge-2 | 22.6     | 18.9     | **23.5** |
|                                 | Rouge-L | 35.1     | **37.6** | 36.0     |
| E2ENLG                          | Rouge-1 | 41.1     | 40.7     | **42.0** |
|                                 | Rouge-2 | 26.3     | 24.2     | **29.0** |
|                                 | Rouge-L | 38.8     | **42.1** | 41.8     |
| WebNLG                          | Rouge-1 | 52.1     | 52.0     | **54.5** |
|                                 | Rouge-2 | 23.9     | 24.6     | **26.8** |
|                                 | Rouge-L | 45.2     | **47.8** | 49.3     |
| Average                         |         | 38.1     | 37.7     | **40.3** |
| **Closed-Book QA**              |         |          |          |          |
| ARC-c                           | EM      | 51.7     | 50.4     | **52.9** |
| ARC-e                           | EM      | 69.7     | 65.7     | **70.3** |
| NQ                              | EM      | 17.3     | 16.1     | **23.5** |
| TQA                             | EM      | **54.5** | 53.9     | 54.0     |
| Average                         | EM      | 48.3     | 46.5     | **50.2** |
| **Big-Bench Hard (BBH)**        |         |          |          |          |
| Boolean Expressions             | EM      | 55.1     | 53.0     | **57.3** |
| Causal Judgement                | EM      | 57.6     | 51.1     | **57.9** |
| Date Understanding              | EM      | **31.0** | 29.3     | 30.7     |
| Disambiguation                  | EM      | 46.6     | 47.2     | **49.3** |
| Penguins in a Table             | EM      | 41.4     | 39.8     | **45.0** |
| Reasoning about Colored Objects | EM      | 35.2     | **37.5** | 33.7     |
| Ruin Names                      | EM      | 19.9     | 19.3     | **21.2** |
| Average                         | EM      | 38.4     | 33.2     | **42.2** |

**References**

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

Reviewer comment (Novelty):
The related work in Section 2.2 can be more clearly explained. Particularly, it is claimed that the "arithmetic operation-based fusion" suffers from "identity confusion among multiple LoRAs". This issue needs to be clarified. How does the proposed approach fix this issue? Also, the details of the "reference tuning-based fusion" method are unclear. Is the approach from Gu et al., 2023 comparable to this work? If so, why is this approach not compared against them?

Author response:
The problem of "identity confusion among multiple LoRAs" in arithmetic operation-based fusion is rooted in our 

* *Observation 1 (in Section 3.1): "Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity."*

In the context of the V&L domain, directly merging multiple LoRAs can result in the model producing images with unclear meaning, as depicted in Figure 3 of the main text. Conversely, fusing LoRAs with weight normalization can lead to the model losing the distinctive characteristics of individual LoRAs, which is undesirable.

In the NLP domain, we trained a LoRA on a combination of ANLI-RA, ANLI-R2, and QNLI datasets and observed that using five or more LoRAs caused the model's output to become chaotic, repeating words like 'like, like, like...'. This demonstrates that directly fusing multiple LoRAs in NLP can negatively impact generative abilities. Additionally, when we applied weight normalization to combine LoRAs trained on five datasets (ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI), we noticed a decrease in performance on four of the datasets, as shown in the table below:

| Model         | ANLI-R1       | ANLI-R2       | ANLI-R3 | QNLI          | WNLI          | Avg           |
| ------------- | ------------- | ------------- | ------- | ------------- | ------------- | ------------- |
| Single LoRA   | 80.32         | 79.02         | 75.92   | 78.62         | 74.32         | 77.64         |
| Simple Fusion | 79.32 (**↓**) | 78.88 (**↓**) | 76.42   | 78.06 (**↓**) | 69.98 (**↓**) | 76.53 (**↓**) |

***How does the proposed approach fix this issue?***

Our proposed approach is primarily based on 

* *Observation 2 (Section 3.1), where we found that different layers of LoRA represent different characteristics.* 

In the V&L domain, as illustrated in Figure 2 of the main text, different layers generate different features for target objects. In the NLP domain, as supported by our experiments, different layers of LoRA acquire distinct feature representations, as shown in the table below:

|           | ANLI-R1   | ANLI-R2   | QNLI      |
| --------- | --------- | --------- | --------- |
| Full LoRA | 81.65     | 80.03     | 76.42     |
| 0%-20%    | 78.72     | 78.35     | **78.14** |
| 20%-40%   | 76.10     | 77.96     | 77.85     |
| 40%-60%   | 76.95     | **81.47** | 74.57     |
| 60%-80%   | 77.25     | 78.19     | 75.71     |
| 80%-100%  | **82.59** | 77.91     | 75.48     |

Our approach leverages this observation to achieve more dynamic fusion. We allocate higher weights to the characteristics needed within each layer and reduce weights for unnecessary features, effectively addressing the issue of identity confusion among multiple LoRAs. This allows us to achieve more tailored and effective fusion within limited parameter capacity.

For instance, in the V&L domain, as depicted in Figure 9 of the main text, our MoLE effectively preserves the characteristics of individual objects. In the NLP domain, as shown in Table 2 of the main text, our MoLE demonstrates comprehensive performance improvements compared to simple direct fusion. These experiments collectively demonstrate that our MoLE can dynamically select and weight LoRA features through hierarchical control, resulting in more effective and meaningful fusion.

Reviewer comment (Reproducibility):
Training method: Section 3.5 says that only the gating parameters were trained, and the others were frozen. Does this mean the LoRA parameters were first individually trained and then kept frozen while the gating parameters were being trained? The current text implies that the LoRA parameters were never trained.

Author response:
**A2.** The reference tuning-based fusion method predominantly centers on the V&L domain and necessitates the incorporation of supplementary information, such as deliberately designed object position masks. This methodology introduces a kind of prior knowledge that our model does not depend on. Consequently, conducting a direct comparison between our model and this method may not be entirely equitable, given the disparate assumptions and information inputs involved.

Reviewer comment (Experiments):
Experiments: How were the number of experts chosen for both the experiments?

Author response:
**A3.** Your understanding is correct. The purpose of our paper is to explore a better way to merge LoRA candidates. Here, "LoRA candidates" refer to LoRA models that have been independently trained on different datasets for distinct purposes. After obtaining these LoRA candidates, we then freeze the parameters of them and train the gating function to merge them.

Reviewer comment (Experiments):
Results: The results in Table 1 are unclear. What is the difference between LoRA expert 1 vs. expert 2 in the image alignment experiments? Is it the number of experts used? Why are these settings not shown for text alignment?

Author response:
**A6.**

* For V&L, we follow the common diffusion multi-concept composition setting$^{[1][2]}$, which usually compose 2~3 concepts into a new multi-concepts image.
* For NLP domain, We determined the number of LoRA candidates based on the dataset sizes of various subdomains within the FLAN dataset. Specifically, within the NLI-related datasets in FLAN, there are several sub-datasets, including ANLI (R1-R3), RTE, CB, WNLI, QNLI, MNLI, among others. Therefore, we selected a subset (5~6) of these as LoRA candidates, while the remainder served as the training set. Given the limited number of LoRA concepts, we conduct new experiments by extending the number of experts from 5 to 128, the following  results are shown in **A7**.

Reviewer comment (Experiments):
Each of the experts is trained on one NLI dataset, and the mixture is evaluated on a mixture of NLI datasets. These are all essentially the same task. What are the experts expected to learn differently in these tasks? It would make more sense to build task specific experts and generalize to new tasks.

Author response:
**A4.** We apologize for any ambiguity in our explanation. In the rows pertaining to image alignment, the expert indices (e.g., expert 1, expert 2) indicate which expert's training images were utilized to calculate the visual similarity with the final generated image. For example, expert 1 might correspond to 'Dog' while expert 2 corresponds to 'Wolf'. The 'text-alignment score' signifies the similarity between the final generated image and the text prompt used to create it. 

To clarify further, we employ a specific prompt (e.g., **t**: "a photo of a [𝑉1] dog wearing [𝑉3] sunglasses, with a [𝑉2] cat beside") to generate the final image **F**, which includes elements such as [𝑉1] dog, [𝑉2] cat, and [𝑉3] sunglasses. We determine the similarity between **F** and **t** using CLIP, resulting in the text-alignment score. Additionally, we compute the similarity between **F** and the images corresponding to [𝑉1] dog, [𝑉2] cat, and [𝑉3] sunglasses (corresponding to LoRA expert 1, 2, and 3 as mentioned in the table) to calculate the image-alignment score.

We acknowledge the need for clarification, and we will ensure that this ambiguity is addressed comprehensively in the final version of our paper.

Reviewer comment (Evaluation):
In Table 1, results vary across settings (e.g.: robot+dog+clock vs. table+can+teapot) and the baselines are better in some settings. Do you have any insights into why MoLE works better in some settings and not all?

Author response:
**A8**. As shown in following Table, there is usually a trade-off between text-alignment and image-alignment$^{[1][2]}$ on the multi-concepts fusion in V & L domain (The visualization of this trade-off  can be seen in the Figure.12 in the supp). High image-alignment leads to a decrease in text-alignment where sample generations have less variance and are close to input target images. As a result, our MoLE may not perform better in both image and text sub alignment. 

| Training Steps | Text Alignment Similarity | Image Alignment Similarity |
| -------------- | ------------------------- | -------------------------- |
| 500            | 0.83                      | 0.68                       |
| 1000           | 0.81                      | 0.73                       |
| 1500           | 0.80                      | 0.75                       |
| 2000           | 0.795                     | 0.76                       |
| 2500           | 0.792                     | 0.77                       |
| 3000           | 0.786                     | 0.774                      |

However, as depicted in Table 1 in the main text, it is worth noting that when evaluating the overall average across all configurations, our MoLE consistently outperforms both in average text alignment and image alignment. This underscores MoLE's capability to achieve superior alignment between text and images. Furthermore, we present additional results in the V&L domain following to further validate MoLE's effectiveness. In these results, it's evident that MoLE achieves the highest average scores in both image alignment and text alignment.

Reviewer comment (Presentation):
In Table 2, the labels under the “Model” column in the “In-Domain” setting must be typos. Instead of MoLE^{r}, MoLE^{c} etc., these should be MoLE^{ar1}, MoLE^{ar2} etc. Can you please confirm?

Author response:
**A5.** Thank you for bringing this to our attention, and we appreciate your diligence in reviewing our work. We sincerely apologize for the typos in Table 2. In the 'In-Domain' setting, the labels under the 'Model' column should indeed be MoLE$^{ar1}$, MoLE$^{ar2}$, MoLE$^{ar3}$, and MoLE$^{q}$, respectively. We acknowledge this oversight, and rest assured, we will rectify these errors in the final version of our paper.

Reviewer comment (Novelty):
Combining MOE with LoRA seems straightforward.

Author response:
We  thank the reviewer for valuable insights. The main motivation behind our work is not simply to integrate Mixture-of-Experts (MoE) with LoRA, but rather to build upon our two observations (Section 3.1). 

* *Observation1: Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity.*

* *Observation2: Different layers of LoRA learn distinct feature representations, collectively forming the overall identity of LoRA.*

Firstly, we recognize that LoRA represents distinct features across different layers. Leveraging this insight, we believe that introducing hierarchical control can enable more flexible and dynamic combinations of LoRA while preserving their individual characteristics—a common challenge faced by the research community. In this context, the MoE serves as a valuable tool to facilitate this hierarchical control, effectively harnessing this property to address the challenging problem of LoRA fusion. **In summary, our primary contribution lies in pioneering the exploration of dissecting and layering LoRA for the first time and utilizing this observation to achieve more dynamic LoRA combinations. MoE is a tool we employ to realize this vision**.

Reviewer comment (Experiments):
No other LoRA merging variants are compared in experiments.

Author response:
In the NLP domain, we conducted comparisons between our MoLE model and the state-of-the-art LoRA merging variant, **LoRAHub** $^{[1]}$ (submitted to ICLR24, too). In the V&L domain, we compared our MoLE with the leading LoRA merging variant, **SVDiff** $^{[2]}$. Our results demonstrate that in both domains, our MoLE consistently achieves superior performance.
To further validate the effectiveness of our MoLE, we have conducted additional experiments, including comparisons with recently-proposed LoRA merging variant baselines.

* **NLP domain**

We conducted extensive experiments across various tasks, including **Translation**, **Struct to Text**, **Closed-Book QA**, and multiple subtasks within the challenging **Big-Bench Hard (BBH)** datasets. Additionally, we introduced a new LoRA merging variant called **PEMs** $^{[3]}$, recently proposed in the field of NLP.

The experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of **3.8** over LoRAHub and outperforms PEMs by a substantial margin of **9.0**.

In the domain of natural language generation tasks (**Translation** and **Struct to Text**), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by **1.5** and PEMs by **2.7**. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of **2.1** over LoRAHub and **2.6** over PEMs. These results highlight the effectiveness of our model in generation task.

| Task                            | Metric  | LoRAHub  | PEMs     | Our      |
| ------------------------------- | ------- | -------- | -------- | -------- |
| **Translation**                 |         |          |          |          |
| WMT '14 En->Fr                  | BLEU    | 27.4     | 25.6     | **29.1** |
| WMT '14 Fr->En                  | BLEU    | 29.4     | 27.1     | **31.3** |
| WMT '16 En->De                  | BLEU    | 24.6     | 24.9     | **27.7** |
| WMT '16 De->En                  | BLEU    | **29.9** | 28       | 29.1     |
| WMT '16 En->Ro                  | BLEU    | 17.7     | 15.2     | **18.9** |
| WMT '16 Ro->En                  | BLEU    | 23.5     | 21.7     | **25.1** |
| Average                         | BLEU    | 25.4     | 24.2     | **26.9** |
| **Struct to Text**              |         |          |          |          |
| CommonGen                       | Rouge-1 | 53.7     | 48.8     | **55.1** |
|                                 | Rouge-2 | **23.1** | 22.4     | 23.1     |
|                                 | Rouge-L | 49.7     | 47.2     | **53.9** |
| DART                            | Rouge-1 | 45.3     | 46.2     | **48.8** |
|                                 | Rouge-2 | 22.6     | 18.9     | **23.5** |
|                                 | Rouge-L | 35.1     | **37.6** | 36.0     |
| E2ENLG                          | Rouge-1 | 41.1     | 40.7     | **42.0** |
|                                 | Rouge-2 | 26.3     | 24.2     | **29.0** |
|                                 | Rouge-L | 38.8     | **42.1** | 41.8     |
| WebNLG                          | Rouge-1 | 52.1     | 52.0     | **54.5** |
|                                 | Rouge-2 | 23.9     | 24.6     | **26.8** |
|                                 | Rouge-L | 45.2     | **47.8** | 49.3     |
| Average                         |         | 38.1     | 37.7     | **40.3** |
| **Closed-Book QA**              |         |          |          |          |
| ARC-c                           | EM      | 51.7     | 50.4     | **52.9** |
| ARC-e                           | EM      | 69.7     | 65.7     | **70.3** |
| NQ                              | EM      | 17.3     | 16.1     | **23.5** |
| TQA                             | EM      | **54.5** | 53.9     | 54.0     |
| Average                         | EM      | 48.3     | 46.5     | **50.2** |
| **Big-Bench Hard (BBH)**        |         |          |          |          |
| Boolean Expressions             | EM      | 55.1     | 53.0     | **57.3** |
| Causal Judgement                | EM      | 57.6     | 51.1     | **57.9** |
| Date Understanding              | EM      | **31.0** | 29.3     | 30.7     |
| Disambiguation                  | EM      | 46.6     | 47.2     | **49.3** |
| Penguins in a Table             | EM      | 41.4     | 39.8     | **45.0** |
| Reasoning about Colored Objects | EM      | 35.2     | **37.5** | 33.7     |
| Ruin Names                      | EM      | 19.9     | 19.3     | **21.2** |
| Average                         | EM      | 38.4     | 33.2     | **42.2** |

Reviewer comment (Experiments):
For the gating imbalance, instead of introducing a new loss, can we just increase the temperature? What's the difference in empirical performance?

Author response:
Thank you for the inspiration provided. We conducted relevant experiments on the NLI dataset, and after increasing the temperature (denoted as MoLE$^{T1}$, MoLE$^{T2}$, MoLE$^{T3}$ where T1 < T2 < T3), we observed a decrease in the model's performance with temperature increasing, with it becoming more similar to Simple Fusion in terms of performance. We believe that while increasing the temperature can alleviate the imbalance issue in the gating network, it also limits our MoLE from learning more dynamic LoRA combinations, leading to suboptimal results.

| Model         | ANLI-R1       | ANLI-R2       | ANLI-R3 | QNLI          | WNLI          | Avg           |
| ------------- | ------------- | ------------- | ------- | ------------- | ------------- | ------------- |
| Simple Fusion | 79.32 | 78.88 | 76.42   | 78.06 | 69.98 | 76.53 |
|  MoLE   | **81.49** | **79.38** | **77.63** | **79.52** | **72.31** | **78.07** |
| MoLE$^{T1}$ | 80.52  | 79.27 | 77.30 | 79.11 | 71.07 |  77.45 |
| MoLE$^{T2}$   | 80.01  | 79.03 | 76.33 |  77.81 |  70.37 |  76.71  |
| MoLE$^{T3}$   | 78.50   | 79.20   | 76.07   | 78.02 | 70.00 | 76.35 |

Reviewer comment (Experiments):
From the motivation of the proposed MoLE for hierarchical control, do you think it's better suited for the text-to-image generation task than the NLI task?

Author response:
Our motivation was grounded in the consistent patterns we observed in both the NLP and VL domains. We apologize for the lack of experimental support for our motivations in the main text concerning the NLP domain. Our NLP domain support experiments are as follows:

* *Observation1: Fusing multiple LoRA candidates directly impacts the generative ability of the original model, while applying weight normalization before the fusion process maintains this generative capacity.*

We combined multiple LoRAs with the FLAN-T5 model. When using five or more LoRAs, the model's output became chaotic, repeating words like 'like like like...'. This shows that directly fusing multiple LoRAs in NLP can affect generative abilities. Additionally, we applied weight normalization to combine LoRAs trained on five datasets (ANLI-R1, ANLI-R2, ANLI-R3, QNLI, and WNLI). As shown in the following table, we observed a decrease in performance for the fused LoRA on four of the datasets. This indicates that while weight normalization preserves generative capacity, it also impacts the intrinsic properties of LoRA.


| Model         | ANLI-R1       | ANLI-R2       | ANLI-R3 | QNLI          | WNLI          | Avg           |
| ------------- | ------------- | ------------- | ------- | ------------- | ------------- | ------------- |
| Single LoRA   | 80.32         | 79.02         | 75.92   | 78.62         | 74.32         | 77.64         |
| Simple Fusion | 79.32 (**↓**) | 78.88 (**↓**) | 76.42   | 78.06 (**↓**) | 69.98 (**↓**) | 76.53 (**↓**) |


* *Observation2: Different layers of LoRA learn distinct feature representations, collectively forming the overall identity of LoRA.*

We trained a LoRA on a combination of ANLI-RA, ANLI-R2, and QNLI datasets, as shown in the table below (Visualized results can be found in Figure 11 of the PDF). We observed significant performance variations across different layers of this LoRA when evaluated on these sub-datasets. Specifically, the 0%\~20% layers performed best on QNLI, the 40%\~60% layers excelled on ANLI-R2, and the 80%\~100% layers outperformed others on ANLI-R1. This reinforces the idea that in the NLP domain, different layers of LoRA acquire distinct feature representations.

|           | ANLI-R1   | ANLI-R2   | QNLI      |
| --------- | --------- | --------- | --------- |
| Full LoRA | 81.65     | 80.03     | 76.42     |
| 0%-20%    | 78.72     | 78.35     | **78.14** |
| 20%-40%   | 76.10     | 77.96     | 77.85     |
| 40%-60%   | 76.95     | **81.47** | 74.57     |
| 60%-80%   | 77.25     | 78.19     | 75.71     |
| 80%-100%  | **82.59** | 77.91     | 75.48     |

Based on the two sets of experiments presented above, we can conclude that the motivations we proposed in Section 3.1 hold true in the NLP domain as well. We will include the supporting experiments in the NLP domain mentioned above in the final version to substantiate our observations

Reviewer comment (Evaluation):
For the NLP domain the evaluation is done only for one classification task (NLI) and no generative task (e.g., summarization or translation) is evaluated. Analogous to vision domain, it would be great to see effect would MOLE bring in during generation.

Author response:
We thank the Reviewer for the insightful suggestions. Regarding the questions 2 raised:

-------
> **Q2**: Did authors conduct experiments on the task of machine translation or summarization or any other text generation task?

**A2**. We conducted extensive experiments across various tasks, including **Translation**, **Struct to Text**, **Closed-Book QA**, and multiple subtasks within the challenging **Big-Bench Hard (BBH)** datasets. Additionally, we introduced a new LoRA merging variant called **PEMs** $^{[1]}$, recently proposed in the field of NLP.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of **3.8** over LoRAHub and outperforms PEMs by a substantial margin of **9.0**.

In the domain of natural language generation tasks (**Translation** and **Struct to Text**), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by **1.5** and PEMs by **2.7**. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of **2.1** over LoRAHub and **2.6** over PEMs. These results highlight the effectiveness of our model in generation task.



| Task                            | Metric  | LoRAHub  | PEMs     | Our      |
| ------------------------------- | ------- | -------- | -------- | -------- |
| **Translation**                 |         |          |          |          |
| WMT '14 En->Fr                  | BLEU    | 27.4     | 25.6     | **29.1** |
| WMT '14 Fr->En                  | BLEU    | 29.4     | 27.1     | **31.3** |
| WMT '16 En->De                  | BLEU    | 24.6     | 24.9     | **27.7** |
| WMT '16 De->En                  | BLEU    | **29.9** | 28       | 29.1     |
| WMT '16 En->Ro                  | BLEU    | 17.7     | 15.2     | **18.9** |
| WMT '16 Ro->En                  | BLEU    | 23.5     | 21.7     | **25.1** |
| Average                         | BLEU    | 25.4     | 24.2     | **26.9** |
| **Struct to Text**              |         |          |          |          |
| CommonGen                       | Rouge-1 | 53.7     | 48.8     | **55.1** |
|                                 | Rouge-2 | **23.1** | 22.4     | 23.1     |
|                                 | Rouge-L | 49.7     | 47.2     | **53.9** |
| DART                            | Rouge-1 | 45.3     | 46.2     | **48.8** |
|                                 | Rouge-2 | 22.6     | 18.9     | **23.5** |
|                                 | Rouge-L | 35.1     | **37.6** | 36.0     |
| E2ENLG                          | Rouge-1 | 41.1     | 40.7     | **42.0** |
|                                 | Rouge-2 | 26.3     | 24.2     | **29.0** |
|                                 | Rouge-L | 38.8     | **42.1** | 41.8     |
| WebNLG                          | Rouge-1 | 52.1     | 52.0     | **54.5** |
|                                 | Rouge-2 | 23.9     | 24.6     | **26.8** |
|                                 | Rouge-L | 45.2     | **47.8** | 49.3     |
| Average                         |         | 38.1     | 37.7     | **40.3** |
| **Closed-Book QA**              |         |          |          |          |
| ARC-c                           | EM      | 51.7     | 50.4     | **52.9** |
| ARC-e                           | EM      | 69.7     | 65.7     | **70.3** |
| NQ                              | EM      | 17.3     | 16.1     | **23.5** |
| TQA                             | EM      | **54.5** | 53.9     | 54.0     |
| Average                         | EM      | 48.3     | 46.5     | **50.2** |
| **Big-Bench Hard (BBH)**        |         |          |          |          |
| Boolean Expressions             | EM      | 55.1     | 53.0     | **57.3** |
| Causal Judgement                | EM      | 57.6     | 51.1     | **57.9** |
| Date Understanding              | EM      | **31.0** | 29.3     | 30.7     |
| Disambiguation                  | EM      | 46.6     | 47.2     | **49.3** |
| Penguins in a Table             | EM      | 41.4     | 39.8     | **45.0** |
| Reasoning about Colored Objects | EM      | 35.2     | **37.5** | 33.7     |
| Ruin Names                      | EM      | 19.9     | 19.3     | **21.2** |
| Average                         | EM      | 38.4     | 33.2     | **42.2** |

**References**

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

------
We want to express our sincere gratitude for your review. We apologize for the delayed response, as we have been dedicating an extended amount of time to conducting experiments, which has kept you waiting. Please let us know if any of your points were not addressed properly, or if you have any additional questions.

Reviewer comment (Experiments):
Did authors conduct experiments on the task of machine translation or summarization or any other text generation task?

Author response:
We thank the Reviewer for the insightful suggestions. Regarding the questions 2 raised:

-------
> **Q2**: Did authors conduct experiments on the task of machine translation or summarization or any other text generation task?

**A2**. We conducted extensive experiments across various tasks, including **Translation**, **Struct to Text**, **Closed-Book QA**, and multiple subtasks within the challenging **Big-Bench Hard (BBH)** datasets. Additionally, we introduced a new LoRA merging variant called **PEMs** $^{[1]}$, recently proposed in the field of NLP.

The corresponding experimental results are summarized in the following table. In summary, our MoLE surpasses state-of-the-art LoRA merging variants on four distinct datasets and tasks, showcasing robust performance. With a notable highlight on the BBH dataset, our MoLE achieves an average performance improvement of **3.8** over LoRAHub and outperforms PEMs by a substantial margin of **9.0**.

In the domain of natural language generation tasks (**Translation** and **Struct to Text**), our MoLE consistently demonstrates superior average performance in the Translation task set, surpassing LoRAHub by **1.5** and PEMs by **2.7**. Similarly, within the Struct to Text task set, our model achieves an average performance advantage of **2.1** over LoRAHub and **2.6** over PEMs. These results highlight the effectiveness of our model in generation task.



| Task                            | Metric  | LoRAHub  | PEMs     | Our      |
| ------------------------------- | ------- | -------- | -------- | -------- |
| **Translation**                 |         |          |          |          |
| WMT '14 En->Fr                  | BLEU    | 27.4     | 25.6     | **29.1** |
| WMT '14 Fr->En                  | BLEU    | 29.4     | 27.1     | **31.3** |
| WMT '16 En->De                  | BLEU    | 24.6     | 24.9     | **27.7** |
| WMT '16 De->En                  | BLEU    | **29.9** | 28       | 29.1     |
| WMT '16 En->Ro                  | BLEU    | 17.7     | 15.2     | **18.9** |
| WMT '16 Ro->En                  | BLEU    | 23.5     | 21.7     | **25.1** |
| Average                         | BLEU    | 25.4     | 24.2     | **26.9** |
| **Struct to Text**              |         |          |          |          |
| CommonGen                       | Rouge-1 | 53.7     | 48.8     | **55.1** |
|                                 | Rouge-2 | **23.1** | 22.4     | 23.1     |
|                                 | Rouge-L | 49.7     | 47.2     | **53.9** |
| DART                            | Rouge-1 | 45.3     | 46.2     | **48.8** |
|                                 | Rouge-2 | 22.6     | 18.9     | **23.5** |
|                                 | Rouge-L | 35.1     | **37.6** | 36.0     |
| E2ENLG                          | Rouge-1 | 41.1     | 40.7     | **42.0** |
|                                 | Rouge-2 | 26.3     | 24.2     | **29.0** |
|                                 | Rouge-L | 38.8     | **42.1** | 41.8     |
| WebNLG                          | Rouge-1 | 52.1     | 52.0     | **54.5** |
|                                 | Rouge-2 | 23.9     | 24.6     | **26.8** |
|                                 | Rouge-L | 45.2     | **47.8** | 49.3     |
| Average                         |         | 38.1     | 37.7     | **40.3** |
| **Closed-Book QA**              |         |          |          |          |
| ARC-c                           | EM      | 51.7     | 50.4     | **52.9** |
| ARC-e                           | EM      | 69.7     | 65.7     | **70.3** |
| NQ                              | EM      | 17.3     | 16.1     | **23.5** |
| TQA                             | EM      | **54.5** | 53.9     | 54.0     |
| Average                         | EM      | 48.3     | 46.5     | **50.2** |
| **Big-Bench Hard (BBH)**        |         |          |          |          |
| Boolean Expressions             | EM      | 55.1     | 53.0     | **57.3** |
| Causal Judgement                | EM      | 57.6     | 51.1     | **57.9** |
| Date Understanding              | EM      | **31.0** | 29.3     | 30.7     |
| Disambiguation                  | EM      | 46.6     | 47.2     | **49.3** |
| Penguins in a Table             | EM      | 41.4     | 39.8     | **45.0** |
| Reasoning about Colored Objects | EM      | 35.2     | **37.5** | 33.7     |
| Ruin Names                      | EM      | 19.9     | 19.3     | **21.2** |
| Average                         | EM      | 38.4     | 33.2     | **42.2** |

**References**

[1] Zhang, J., Chen, S., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.

------
We want to express our sincere gratitude for your review. We apologize for the delayed response, as we have been dedicating an extended amount of time to conducting experiments, which has kept you waiting. Please let us know if any of your points were not addressed properly, or if you have any additional questions.