Paper: Enhancing Parameter Efficiency in Summarization via Expertise Separation (ICLR-2024)

Reviewer comment (Experiments):
As this paper mentions the proposed method is **parameter-efficient**, experiments should include more than one base model to confirm this, otherwise we are not sure if this method only works on BART. For example, this paper can add PEGASUS as [1] used.

Author response:
Thanks for your thoughtful consideration. In our Appendix A.4, we include the comparison with not only BART-large, but also BART-base and PEGASUS. The results show the generalization ability of our framework that does not depend on specific model scales or architectures. We choose the BART-large structure in our main experiment due to its better performance.

Reviewer comment (Theory):
Allocating each deputy to a single dataset doesn’t sound too practical and scalable as the number of datasets increases. One can question that if many datasets are similar, why are we not sharing similar deputies to many applications that are similar/related to each other?

Author response:
We apologize for any confusion caused. Indeed, you are absolutely correct in noting that similar applications should share similar deputy experts, which aligns perfectly with our motivation for our MoeSumm structure. In practice, we do not allocate a specific deputy to a single dataset. Instead, we utilize shared multiple deputy experts which are dynamically selected based on the context and dataset category. We have revised and emphasized this aspect in Section 4 for clearer understanding.

Reviewer comment (Experiments):
Also regarding the loss: I personally think one explanation for the max-margin loss is to battle with the imbalance, at the same time train the deputy to “spread-out” and learn all diverse data being fed. As a result, my hypothesis is with the current design, the imbalance loss from, say Gshard (and other methods could be used too in the RELATED WORK point below). Would be nice to study the ablations at some point.

Author response:
A3: Thanks for listing the interesting references about the imbalance issue in MoE. However, we would like to clarify that our max-margin loss is not to balance between deputy experts. Instead, it is an adapted loss for hierarchical MoE structure, to differentiate between general ability in main expert and specialized ability in deputy experts. Hence, the listed works on imbalance loss can not replace our max-margin loss. Still, we are happy to include these references in our revision.

Reviewer comment (Experiments):
Human evaluation probably needs further clarification and only 3 of them do not seem convincing enough. Also, how were their employments made (see ethics section as well), and why only PhD students were chosen?

Author response:
We ensure high-quality annotations by hiring well-educated PhD students, specifically selecting those with backgrounds in computer science, chemistry, and biomedicine to guarantee diversity. Each case was compensated at \$ 0.01\$, leading to a total payment of \$6 per student.

Reviewer comment (Evaluation):
BertScore is a decent choice, but I suggest since you already used GPT3.5, that you could use GPT3.5 or 4 to do the comparison as well, by having an instruction and ask the model to compare different models based on, e.g. 3 criteria in Table 2, then summarize the results. I think for now that would offer a very convincing evaluation.

Author response:
Thanks to your advice, we conduct an evaluation with the latest GPT4. The instruction is:
> Score summary1, summary2, and summary3 given the article in respect to fluency, with one (\*) to three stars (\***), where one star means bad and three stars means good. Only return the stars.

|         | Succ | Inform | Flu  |
|---|----|--|--|
| BART-mix| 2.32 | 2.36   | 2.68 |
| MoeSumm | 2.63 | 2.66   | 2.89 |
| GPT-3.5 | 2.81 | 2.90   | 2.95 |

The ranking aligns with our human assessment, affirming the reliability and effectiveness of both evaluation methods and showing our model's advantage.

Reviewer comment (Reproducibility):
In Table3, “MoESumm w/o DI”, how did you construct the max-margin loss without DI information?

Author response:
By removing the dataset information in the selector, the dataset-aware selector degrades into a naive selector. That is, the $W_e$ matrix for $e$ dataset is changed to $W$, which is the same for all the datasets. 
Though the dataset information is removed from the selector, we still have the main and deputy experts, thus, the max-margin loss still works in differentiating these experts.

Reviewer comment (Evaluation):
The proposed method presents some scaling challenges, which are partially discussed in Appendix A.7, stating that the number of expert model parameters scale with the number of experts and datasets. Furthermore, there is a concatenation of parameters and activations in equations 6 and 7, which would result in more parameters. Consequently, when using experts, the concatenation of features makes MoeSumm a larger model than the BART-indiv baseline, which could explain the advantage of MoeSumm in the in-domain setting. Thus, it would be very informative to have a table with actual parameter count for each setting, especially during training (frequently the most important bottleneck).

Author response:
A1: Thanks for your suggestion, below we show the parameter table and we also include it in the revision in Appendix 9:


| Model Name            | BART-large | Prefix | MoeSumm (3 datasets) | MoeSumm (4 datasets) | MoeSumm (5 datasets) | MoeSumm (6 datasets) |
|-----------------------|------------|--------|----------------------|----------------------|----------------------|----------------------|
| Parameter Number      | 387.46M    | 449.42M (+16%) | 411.77M (+6%) | 411.84M (+6%) | 411.91M (+6%) | 411.98M (+6%) |




It can be seen that further scaling up the training data does not significantly increase the number of parameters. We have included this table in the revised version.

Reviewer comment (Experiments):
Previous work that approaches parameter-efficient summarization by separating domain-specific concerns is not compared. FactorSum (Fonseca et al., 2022) also uses a BART backbone to capture general importance and uses a separate optimization procedure for domain-specific factors such as summary conciseness. It achieves superior performance (measured by ROUGE) than MoeSumm on arXiv and PubMed using less parameters (BART-base). In contrast, MoeSumm explores a wider range of domain-adaptation settings.

Author response:
Thanks for bringing this great work to our attention, and we’ve included it in the revision! In the meantime, we would like to highlight the essential differences between our work: 

In FactorSum, the BART backbone model is trained on a single target dataset to acquire general summarization skills, where ‘general’ refers to summarization without specific consideration for their length. Consequently, a separate model is trained for each dataset to accommodate the different summarization requirements.


In contrast, in our setting, the general summarization ability is not limited to one dataset and its length information, but it is learned from different datasets. Take Figure 5b in the paper for example, our specialized ability includes the understanding of the professional terms in a specific domain.

Reviewer comment (Evaluation):
While I appreciate the analysis of dataset features in Section 5.2, item 3, more detailed analysis of the weight of each feature would be informative. Especially with respect to *summary length*, which is known to heavily affect ROUGE (Sun et al., 2019).

Author response:
The summary length is indeed an important factor when evaluating the summary performance. As shown in Figure 5a, we can see that MoeSumm w/o deputy experts lacks information regarding the target length, while MoeSumm efficiently halts the generation process to produce an optimized summary length. The use of deputy experts is also linked to the length of summaries. For instance, our findings show that MoeSumm when paired with Deputy Expert #1 typically produces shorter sentences, averaging around 15 words, whereas MoeSumm working with Deputy Expert #3 tends to create longer sentences, averaging about 37 words.

Reviewer comment (Evaluation):
Why do you think the zero-shot performance of MoeSumm is significantly better than BART-mix on some datasets (arXiv, MultiNews) but not others (Gigaword, Billsum)?

Author response:
Thank you for your insightful question. We believe that MoeSumm's effectiveness is attributed to reusing summarization skills from in-domain training datasets. For arXiv and MultiNews, which feature long, specialized content, skills learned from similar datasets like PubMed and CNN/DM are applicable. Conversely, Gigaword and Billsum present unique challenges due to their brevity and legislative content, not covered in our training. Following this direction, we draw the performance on Gigaword as training datasets get larger. As shown in Appendix A.5, MoeSumm's performance on Gigaword notably improves (by 7% in ROUGE-1 scores) when trained with similar datasets such as AESLC and Reddit.

Reviewer comment (Reproducibility):
What are the parameter counts for each model variant during training/inference? (see weakness comment above)

Author response:
A1: Thanks for your suggestion, below we show the parameter table and we also include it in the revision in Appendix 9:


| Model Name            | BART-large | Prefix | MoeSumm (3 datasets) | MoeSumm (4 datasets) | MoeSumm (5 datasets) | MoeSumm (6 datasets) |
|-----------------------|------------|--------|----------------------|----------------------|----------------------|----------------------|
| Parameter Number      | 387.46M    | 449.42M (+16%) | 411.77M (+6%) | 411.84M (+6%) | 411.91M (+6%) | 411.98M (+6%) |




It can be seen that further scaling up the training data does not significantly increase the number of parameters. We have included this table in the revised version.

Reviewer comment (Evaluation):
The efficiency is not the superiority of the paper since many paper with few-shot setting directly tune the models. MoE models are usually good in the performance but worse in the efficiency, similar with the ensemble methods.

Author response:
*Q2: MoE models are worse in efficiency in comparison with existing few-shot baselines that directly finetune the model.*

A2: Firstly, we want to clarify that our model has fewer parameters compared with a number of existing few-shot baselines. As shown in the table, our model has 10% less parameters compared with the recent baseline Prefix and 24% less parameters with Lightweight.

Secondly, our model is different from existing MoE models. Our MoeSumm is a hierarchical structure with main and deputy experts. The general summarization ability for various domains is shared to achieve parameter efficiency.

| Model Name            | Parameter Number         |
|-----------------------|--------------------------|
| BART-large            | 387.46M                  |
| Prefix                | 449.42M (+16%)           |
| Lightweight           | 508.62M (+30%)           |
| MoeSumm  | 411.77M (+6%)            |

------
*W2: Reducing 10% parameters may not be significant*

A2: Thank you for your feedback. We wish to highlight the parameter efficiency of our model compared with a number of baselines. It saves 10% more parameters than [Prefix, ACL 2021], 24% more than [Lightweight, SIGIR 2022], and 34% more than [Efficient framework for low-resource abstractive summarization by meta-transfer learning and pointer-generator networks, 2023]. This significant reduction in parameters, while maintaining performance, marks a noteworthy achievement in summarization research.

Moreover, we would like to highlight that our key contribution lies in the novel design of a hierarchical mixture of expert structure with expertise separation. This design enables MoeSumm to utilize a main expert for general summarization capabilities while employing deputy experts for specific summarization tasks. This hierarchical approach distinguishes our work from existing MoE methods in the field.

Reviewer comment (Novelty):
The contribution is relatively minor. It is suggested to try some alternative designs for justifying the proposed approach.

Author response:
*Q3: Contributions of the paper*

A3: The contribution of the paper can be summarized into three aspects:
(1) We propose MoeSumm, a parameter-efficient summarization model that is  applicable to a variety of in-domain summarization tasks and is well-suited for out-of-domain few-shot and zero-shot summarization scenarios. The model's parameter efficiency is ensured by the shared general summarization ability. (2) To achieve the above goal, we design MoeSumm with a mixture-of-expert structure and a max-margin loss to distinguish the roles of the different experts. (3) Experimental results demonstrate that   MoeSumm brings substantial improvements over strong baselines in both in-domain and out-of-domain scenarios across benchmark datasets.

Reviewer comment (Presentation):
The titles in the paper and system are different, i.e., "Enhancing Parameter Efficiency in Summarization via Expertise Separation" and "Flexible and Adaptable Summarization via Expertise Separation". It is suggested to select one and revise the paper accordingly.

Author response:
*Q4: The title in the system and the title in the paper are not consistent.*

A4: Thank you for the reminder. We’ll keep the title in the paper ‘Flexible and Adaptable Summarization via Expertise Separation’.