Paper: Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR-2024)

Reviewer comment (Reproducibility):
Experts permutation alignment (then computing similarity based on routes) is a big part of the paper, but the details are a bit lost on me. It could be worth adding more to 3.1, covering the basics of how the different possible permutations are searched.

Author response:
**[W1. Details of Experts Permutation Alignment]**

Thanks for this very constructive comment about “covering the basics of how the different possible permutations are searched”. Searching optimal permutation can be formulated as a “linear assignment problem” and efficiently solved, specifically:

-   In detail, the permutation matrix is initially defined as a square matrix with each row and column containing precisely one element valued at $1$, while all remaining elements are $0$. Considering an expert with two distinct weight matrices $\mathtt{W}\_{\text {in}}$ and $\mathtt{W}\_{\text {out}}$, the rearrangement of these weight matrices under permutation is represented as $\mathtt{W}\_{\text{out}}\mathtt P^{\mathrm T}$ and $\mathtt P\mathtt{W}\_{\text {in}}$, respectively, for a chosen permutation matrix $\mathtt P$.

-   To determine the optimal permutation matrix, our strategy is to minimize the L2 norm of the difference between the permuted matrices. This involves an optimization method formulated as:
    $$\mathrm{argmin}\_{\mathtt P} \left\|\mathrm{vec}([\mathtt W\_{\text {in}}^{(1)}, \mathtt W\_{\text {out}}^{(1)}]) - \mathrm{vec}([\mathtt P\mathtt W\_{\text {in}}^{(2)}, \mathtt W\_{\text {out}}^{(2)}\mathtt P^{\text T}])\right\|^2 = \mathrm {argmax}\_{\mathtt P} \left\langle \mathtt W\_{\text {in}}^{(1)}, \mathtt P\mathtt W\_{\text {in}}^{(2)} \right\rangle\_{\text F} + \left\langle \mathtt W\_{\text {out}}^{(1)}, \mathtt W\_{\text {out}}^{(2)}\mathtt P^{\text T}\right\rangle\_{\text F}$$ 

The process of optimizing this is known as a 'linear assignment problem' (LAP), for which the Hungarian Algorithm ([R1]) provides an effective and feasible solution. More details are included in the last paragraph of Appendix 2, and the Python-style pseudo-code is provided in Appendix 3, in our revised paper.

Reviewer comment (Theory):
In 3.1, it *seems like there's a chicken and egg problem -- we need alignment to know which experts are more similar* but we need to know which expert is the "reference" to re-align. How is that resolved?

Author response:
**[W2. Experts Permutation and Grouping]**

Thank you for your insightful question. We address your concerns from several aspects: 

-   Your point about the “chick and egg problem” is interesting, but factually we do not “need alignment to know which experts are more similar”. This is because we compute pairwise similarity among the experts only based on the router’s output logits while permuting expert weight matrices does not influence it. Therefore, it is appropriate to conduct permutation alignment of experts before calculating their similarities.
-   Yes, aligning experts within a group together is optimal for merging. However, computing the optimal solution for the permutation alignment of all experts simultaneously would be extremely time-consuming. Therefore, we align all experts in a single SMoE layer with the first expert. 
-   To further address your concerns, we conduct additional experiments to compare the performance of within-layer aligned (ours) M-SMoE and within-group aligned M-SMoE. As shown in Table R1, the performance difference between the two is negligible. 

Table R1: M-SMoE performance on the COPA and MultiRC tasks with the *switch-base-32* model.

| Method                        | COPA  | MultiRC |
| ----------------------------- | ----- | ------- |
| Within-layer Alignment (ours) | $68.00$ | $75.57$   |
| Within-group Alignment        | $68.00$ | $75.53$   |

Reviewer comment (Presentation):
Nit: the changing underline baseline is visually unappealing. Not sure if there's a way to glue underlines to the bottoms of letters. This is a super-nit of course, doesn't really matter at all.

Author response:
**[Formatting Issues]** 

We appreciate your attention to detail and agree that the visual appeal of the document is important. We have looked into all the formatting issues you mentioned and ensured a more consistent and aesthetically pleasing presentation in our revised paper. Specifically:

*   The issue that “the legend kinda blends in” in Figure 1 has been fixed by giving it “a strong border” and a “clearer background”. Thanks for the constructive suggestion.
*   Thanks for pointing out that “the changing underline baseline is visually unappealing”. We have fixed this issue by replacing the “\underline” latex command with “\uline” provided by the *ulem* package in our revised paper.

Reviewer comment (Theory):
A theoretical proof of either 1. Optimality of their expert merging algorithm or 2. An error bound on either the information loss/performance degradation based on their proposed algorithm would have significantly helped this work.

Author response:
**[W1. Lack of Theoretical Proof]**

Thanks for the suggestions. While the absence of theoretical analysis is noted, it does not diminish the value of our contributions. The idea of providing theoretical proof is appreciated and constructive, however, it falls beyond the scope of our present work. The theoretical justification of model merging is an open research question, and some early explorations include [R1] and [R2]. 

[R1] https://openreview.net/pdf?id=CQsmMYmlP5T

[R2] https://arxiv.org/pdf/1910.05653.pdf

Reviewer comment (Experiments):
An analysis of the computational cost as the number of SMoE layers increases would be helpful.

Author response:
**[W2. Computational Cost Analysis]**

Thank you for your question. We present analysis and empirical results of the “computational cost as the number of SMoE layers increase” of merging. 

-   *<The computational cost analysis for each stage.>* Thanks for the constructive suggestion. We present a detailed computational cost analysis for each stage. Our M-SMoE merging method involves three key processes: expert permutation alignment, expert grouping, and expert weight merging. __Firstly__, expert permutation alignment is executed independently within each SMoE layer, making its computational cost linearly proportional to the number of SMoE layers. __Secondly__, expert grouping involves model inference to assess activation frequency and router logits, followed by calculating pair-wise similarity among experts. Given that the model's inference cost remains constant across different numbers of SMoE layers, due to SMoE's sparse activation computation, the primary factor contributing to a linear increase in computational costs is the similarity computation within each SMoE layer. __Lastly__, the process of expert weight merging, carried out within each SMoE layer, further contributes to this linear escalation in computational costs. __In summary__, while certain components of our method exhibit constant computational demands, the overall cost analysis reveals a linear growth pattern.

*   *<Evaluation of computational costs.>* To further address your concerns, we conduct extra experiments for the computational costs of merging. We evaluate the *switch-base-32* model’s computation time cost of expert permutation alignment, expert grouping, and expert weight merging respectively. We maintained a constant ($24$) total number of Transformer layers while varying the number of SMoE layers. The results shown in Table R1 confirm our analysis, indicating that the primary bottleneck in terms of time cost is rooted in the expert permutation alignment, while the bulk of memory cost is attributed to model inference.

Table R1: Evaluation of time cost and memory cost for each stage of M-SMoE.

| Stage                        | Metric      | 2          | 4          | 6          | 8          | 10          | 12          |
| ---------------------------- | ----------- | ---------- | ---------- | ---------- | ---------- | ----------- | ----------- |
| Expert Permutation Alignment | Time Cost   | $2.35$ min | $4.61$ min | $6.54$ min | $8.40$ min | $10.30$ min | $12.30$ min |
|                              | Memory Cost | $1.23$ GB  | $2.36$ GB  | $3.48$ GB  | $4.61$ GB  | $5.73$ GB   | $6.86$ GB   |
| Expert Grouping              | Time Cost   | $8.03$ s   | $8.21$ s   | $8.22$ s   | $8.26$ s   | $8.20$ s    | $8.24$ s    |
|                              | Memory Cost | $4.19$ GB  | $5.29$ GB  | $6.39$ GB  | $7.48$ GB  | $8.58$ GB   | $9.68$ GB   |
| Expert Weight Merging        | Time Cost   | $23$ ms    | $44$ ms    | $66$ ms    | $87$ ms    | $109$ ms    | $139$ ms    |
|                              | Memory Cost | $1.33$ GB  | $1.83$ GB  | $2.32$ GB  | $2.82$ GB  | $3.31$ GB   | $3.81$ GB   |

Reviewer comment (Theory):
Could you provide some theoretical insights into why the merging of experts should be done before compression since compression might be able to get rid of irrelevant information and make it more convenient to compare experts later for merging?

Author response:
**[Q1. Theoretical Insights of Merging Before Compression]**

Thank you for this excellent question. The reasons lie in several aspects:

-   Our initial intention was to demonstrate that merging can encourage low-rank properties, hence we performed compression after merging. 
-   Moreover, it's important to note that merging will disrupt the sparsity and low-rank patterns established by compression. This is because two compressed experts typically do not exhibit identical patterns. 

Your suggestion was insightful, and it led us to carry out further experiments to compare two approaches: 'merging, then compression' (MC-SMoE, ours) and 'compression, then merging' (CM-SMoE) on the COPA and MultiRC tasks, as detailed in Table R2. The results indicate that MC-SMoE not only surpasses CM-SMoE in terms of task performance metrics but also shows advantages in model size and FLOPs. This advantage is primarily due to the preservation of the compression pattern, which is otherwise disrupted when merging follows compression.

Table R2: Comparison between MC-SMoE and CM-SMoE on the COPA and MultiRC tasks with the *switch-base-32* model.

| Method         | COPA    | MultiRC | Model Size | TFLOPs |
| -------------- | ------- | ------- | ---------- | ------ |
| MC-SMoE (Ours) | $67.00$ | $73.98$ | $381$M     | $3.83$ |
| CM-SMoE        | $64.00$ | $73.63$ | $733$M     | $4.65$ |

Reviewer comment (Theory):
Could you provide some theoretical background for your claim that similar experts would show similar router logits?

Author response:
**[Q2. Theoretical Background of Similarity of Experts and Router Logits]**

-   Thank you for your suggestions. The theoretical justification remains an open research question, and we greatly appreciate your constructive input. We look forward to addressing this aspect in our future work with a thoughtful approach.

-   The insight here is: firstly, SMoE routing tends to stabilize in the early stages of pretraining (which can be shown by Figure 6 in [R3]); therefore, if two experts have similar router logits, it is more likely that they have been trained on similar data, making the experts more alike

[R3] https://openaccess.thecvf.com/content/ICCV2023/papers/Chen_AdaMV-MoE_Adaptive_Multi-Task_Vision_Mixture-of-Experts_ICCV_2023_paper.pd

Reviewer comment (Experiments):
An interesting line of investigation is the long-term scalability of how one could add more experts later during the life-cycle after applying M-SMoE.

Author response:
**[Q3. Long-term Scalability and Adding More Experts Post-M-SMoE]**

This comment is insightful. The issue of 'long-term scalability and the addition of more experts during the life-cycle' has been explored in [R4]. It investigates the dynamic addition of experts complemented by regularized pre-training for extra model capacity. Our paper, meanwhile, focuses on the expert merging method, specifically M-SMoE. A synthesis of these two approaches could effectively address your concerns.

[R4] https://arxiv.org/pdf/2305.12281.pdf

Reviewer comment (Reproducibility):
What is the intuition behind experts permutation alignment? Specifically, how is “alignment” defined? I can understand from performance-wise (Table 6) that alignment is needed. However, I do not understand how two experts are “aligned” using the proposed alignment method. To make the paper self-contained, please include the detailed algorithm used for this.

Author response:
Thanks to reviewer 5S2f for recognizing the importance (“a very important topic”) and motivation (“well-motivated”) behind our work on compressing Mixture-of-Experts (MoE) models. To address reviewer 5S2f’s questions, we provide pointwise responses below.

**[Cons 1. Experts Permutation Alignment]** 

The intuition behind permutation alignment is to pull all the models (i.e. experts within one SMoE layer in our setting) into the same local loss basin, as introduced in the third line of the last paragraph in Section 2 of our paper. Specifically:

-   *<Before alignment.>*  As mentioned in the last three lines in the third paragraph of Section 2 in our paper, the experts in SMoE are basically trained on different data subsets and starting from different initializations, resulting in them converging to different basins of the loss landscape[R3]. There existing loss barrier between each pair of basins, which can be measured via the LMC (Linearly Mode Connected) metric (details are included in the last paragraph of Appendix 2 in our revised paper). Such a barrier hinders the merging of experts before alignment.
-   *<During alignment.>* Alignment permutes the neurons within the expert to minimize the loss barrier between each expert pair while preserving the original functionality. As illustrated in the last few lines in the first paragraph of Section 3.1 in our paper, for a two-layer MLP, both the output channels of its first layer and the input channels of the second layer are permuted via the same order, thus its output embeddings are exactly the same as the one without permutation.
-   *<After alignment.>*  Via expert alignment, the loss barrier between each two experts is significantly reduced, benefiting the further merging process ([R4, R5, R6, R7]).
-   *<Existence of the loss barrier in the SMoE case.>* Moreover, we have carried out additional experiments to validate the presence of the loss barrier in the SMoE scenario. More specifically, for each SMoE layer containing $2N$ experts, we modified the weights of the $i$th expert, $\mathtt{E}\_i$, to become an interpolation of $\lambda \mathtt{E}\_i + (1-\lambda)\mathtt{E}\_{(i+N)\\%N}$ , where $\lambda$ ranges from $0.5$ to $1$. We then compared the performance of the fine-tuned SMoE, respectively with and without permutation alignment, on the COPA task. The performance (accuracy), as detailed in Table R0, corroborates the existence of a loss barrier among the SMoE experts.

Table R0: Accuracy of different values of $\lambda$ on the COPA task of the *switch-base-32* model.
| $\lambda$  | 1.0   | 0.9   | 0.8   | 0.7   | 0.6   | 0.5   |
| ---------- | ----- | ----- | ----- | ----- | ----- | ----- |
| **w.** PA  | 68.00 | 64.00 | 47.00 | 34.00 | 30.00 | 28.00 |
| **wo.** PA | 68.00 | 60.00 | 40.00 | 26.00 | 20.00 | 20.00 |

Thanks for your question, we have enriched the description of the expert permutation alignment procedure and added it to the last of Appendix 2 in our paper. To be more specific:

*   *<The permutation matrix.>* First, we introduce the permutation matrix $\mathtt{P}$, which is a square matrix where each row and column has exactly one element of 1, with all other elements being 0. It perseveres the functionality of the expert, an MLP with two linear layers $\mathtt W_{\text{in}}$ and $\mathtt W_{\text{out}}$.

*   *<The optimization through L2 distance minimization.>* Second, we minimize the L2 distance between two experts to align them. Given two layer weights of an expert, denoted as $\mathtt W\_{\text{in}}$ and $\mathtt W{\text {out}}$, we consider an optimization approach: (More details are included in the last paragraph of Appendix 2 in our revised paper.)
    $$\mathrm{argmin}\_{\mathtt P} \left\|\mathrm{vec}([\mathtt W\_{\text {in}}^{(1)}, \mathtt W\_{\text {out}}^{(1)}]) - \mathrm{vec}([\mathtt P\mathtt W\_{\text {in}}^{(2)}, \mathtt W\_{\text {out}}^{(2)}\mathtt P^{\text T}])\right\|^2 = \mathrm {argmax}\_{\mathtt P} \left\langle \mathtt W\_{\text {in}}^{(1)}, \mathtt P\mathtt W\_{\text {in}}^{(2)} \right\rangle\_{\text F} + \left\langle \mathtt W\_{\text {out}}^{(1)}, \mathtt W\_{\text {out}}^{(2)}\mathtt P^{\text T}\right\rangle\_{\text F}$$

*   *<Solving the LAP optimization.>* Finally, the optimization constitutes a “linear assignment problem” (LAP), which can be efficiently and practically solved by the Hungarian Algorithm ([R8]). 
*   *<Detailed revision.>* All explanations above are included in the sixth paragraph of Appendix 2 in our revised paper. We have further provided the Python-style pseudo code for solving the permutation matrix in Appendix 3 of our revised paper.

[R2] https://arxiv.org/pdf/2306.11222.pdf

[R3] https://arxiv.org/pdf/2110.06296.pdf

[R4] https://openreview.net/pdf?id=CQsmMYmlP5T

[R5] https://arxiv.org/pdf/2211.08403.pdf

[R6] https://arxiv.org/pdf/1910.05653.pdf

[R7] https://arxiv.org/pdf/2212.12042.pdf

[R8] https://web.eecs.umich.edu/~pettie/matching/Kuhn-hungarian-assignment.pdf

Reviewer comment (Reproducibility):
How are the results in Figure 3 computed? From my understanding, a Switch-base-32 model is first fine-tuned on each individual task, and then the activation frequencies are computed. Are the models fine-tuned with the load balancing loss [1]? It seems in Figure 3, loads of different experts are extremely unbalanced.

Author response:
**[Cons 2. Results in Figure 3]** 
Yes, your interpretation that “a switch-base-32 model is first fine-tuned on each individual task, and then the activation frequencies are computed” is correct (which is illustrated in the last two lines of the caption of Figure 3 in our paper). During the fine-tuning process, we did not use load balancing loss, for two reasons:
*   *<Performance-driven concerns.>* Based on our previous investigation, enabling load balancing regularization will degrade the model performance. As shown in Table R1, after carefully sweeping the coefficient of load balance regularization, we can observe a **consistent** performance degradation.

Table R1: Performance evaluation on the COPA and MultiRC tasks of the *switch-base-32* model fine-tuned with different load balancing loss coefficients.

| Coefficient | COPA (Accuracy) | MultiRC (F1-score) |
| ----------- | --------------- | ------------------ |
| $0$         | $68.00$         | $76.19$            |
| $0.001$     | $63.00$         | $74.41$            |
| $0.01$      | $62.00$         | $69.75$            |
| $0.1$       | $58.00$         | $69.09$            |
| $1.0$       | $58.00$         | $68.07$            |
| $10$        | $56.00$         | $59.93$            |

*   *<Redundancy-driven concerns.>* Employing load balancing loss brings more redundancy in SMoE. We assess such redundancy via the performance gap between the fine-tuned model and the model where non-dominant experts are pruned. Specifically, large redundancy indicates the encoded information of pruned experts is also contained in the remaining experts, resulting in a smaller performance drop after pruning. Table R2 demonstrates load balancing regularization will cause more redundancy in SMoEs. (i.e., only $4.00$ performance gap v.s. the original $21.00$)

Table R2: The *switch-base-32* model fine-tuned on the COPA task with the load balancing loss coefficient of $0$ and $10$. The larger performance gap of the model with coefficient=$0$ (ours) between pruning indicates the existence of its less redundancy.

| Model          | Coefficient = $10$ | Coefficient = $0$ |
| -------------- | ---------------- | --------------- |
| Before pruning | $56.00$            | $68.00$           |
| After pruning  | $52.00$            | $47.00$           |
| $\Delta$       | $-4.00$            | $-21.00$          |

Reviewer comment (Evaluation):
How is the stable-rank computed? For example, in a specific layer, the 32 experts are compressed into 6 experts. Do you compute the average stable-rank of the 32 experts as “before” in Figure 4, and the average stable-rank of the 6 experts as “after”?

Author response:
**[Cons 3. Stable Rank Computation]**

>   “For example, in a specific layer, the 32 experts are compressed into 6 experts. Do you compute the average stable rank of the 32 experts as “before” in Figure 4, and the average stable rank of the 6 experts as “after”?” 

No, we compute the stable rank of these 6 dominant experts both "before" and "after" the merging process. Following this, we compute the average of the ratio $\frac{\text{after} - \text{before}}{\text{before}}$ across these 6 experts. A further refinement has been included in Figure 4 of our revised paper.

Reviewer comment (Writing):
I do not fully understand how experts are grouped. It is mentioned that “each non-dominant expert gravitates toward and joins the group led by its most similar dominant expert“. For example, suppose we have two dominant experts \(E_1\) and \(E_2\), then for a non-dominant expert \(E\), do you calculate the similarity of \(E\) with \(E_1\) and \(E_2\), and then assign \(E\) to the more similar one? If this is true, is it possible that nearly all non-dominant experts are assigned to the same group?

Author response:
**[Cons 4. Expert Grouping Process]**

In your example “suppose we have two dominant experts E1 and E2, then for a non-dominant expert E”, yes, we “calculate the similarity of E with E1 and E2, and then assign E to the more similar one”. We recapitulate the grouping procedure and explain the corner case you mentioned as below:

-   *<How to group?>* As illustrated in the third paragraph of Section 3.1 and line 9 of Algorithm 1 of our paper. The grouping process includes: (1) selecting the most frequently activated experts as dominant experts; (2) calculating the pairwise similarity among the experts; and (3) for each non-dominant expert, assigning it to the most similar dominant expert. 
-   *<The corner case.>* Thanks for pointing it out. It is a corner case that “nearly all non-dominant experts are assigned to the same group”, which is reasonable due to the redundancies in SMoE models and confirms our insights. We also provide expert grouping results in Appendix 4.1 of our revised paper and show that this corner case is very rare.

Reviewer comment (Experiments):
The pruning of \(S\) needs more justification. It is mentioned that “the weight columns with the lowest cumulative scores will be removed”. Why are weights pruned according to cumulative importance scores instead of importance scores? The pruning procedure in Appendix A2 also seems ad-hoc. How is this particular pruning schedule chosen?

Author response:
**[Cons 5. Pruning of S and Pruning Schedule]**

**[Details of Cumulative Importance Score]** Thank you for your question. As described in the ninth line of the last paragraph of our paper, it's important to clarify that we prune weight matrix columns based on their 'cumulative importance scores', not just individual 'importance scores' of weights. 

*   *<What is the cumulative importance score?>* The 'cumulative importance score' of a column is a summation of the 'importance scores' of all individual weights in that column. 
*   *<Why cumulative importance score?>* This cumulative approach gives a more comprehensive assessment of the overall significance of a weight matrix column. By considering the collective impact of all weights in a column, we can more accurately determine which columns are less critical to the model's performance and therefore suitable for pruning. In contrast, pruning based on individual weight scores might overlook the collective contribution of weights within a column, potentially leading to less effective pruning decisions. 

**[Details of Pruning Schedule]** We use a cubic schedule of pruning ratio, which is widely applied in many existing methods, such as [R10, R11, R12]. It includes initial and final warmups where the ratio stays constant respectively, while in the middle iterations, the pruning ratio gradually increases following a cubic trend. Details are included in the third paragraph in Appendix 2 of our paper.

**[Experiments - Comparison Between Pruning Schedules]** To address your concerns about the pruning schedule, we conduct additional experiments for comparison between linear, quadratic, and cubic (ours) pruning ratio schedules on the COPA and MultiRC tasks. The results are shown in Table R3, indicating a performance ordering of __cubic (ours) > quadratic > linear__. This is potentially because, in the early stages of pruning, an aggressive pruning schedule is less likely to lose useful information in the weights; while it is the opposite in the later stages of pruning. (This comparison and its analysis have been included in Appendix 1.4 of our revised paper.)

Table R3: MC-SMoE Comparison between {linear, quadratic, cubic (ours)} schedules on the COPA and MultiRC tasks with the *switch-base-32* model.

| Schedule     | COPA    | MultiRC |
| ------------ | ------- | ------- |
| Linear       | $59.00$ | $73.83$ |
| Quadratic    | $61.00$ | $73.92$ |
| Cubic (ours) | $67.00$ | $73.98$ |

Reviewer comment (Experiments):
I would like to further understand the role of knowledge distillation. The authors mention that all the models (including the baselines) in Table 2 use knowledge distillation. Could the authors provide some results of the dense and full-SMoE models without distillation?

Author response:
**[Cons 6. Role of Knowledge Distillation]**

No, the knowledge distillation is not applied to the dense and full-SMoE models, and it is only applied to “all merged and compressed SMoEs” (which is illustrated in the fourth line of the third paragraph of Section 4.3 in our paper). It is to encourage the merged or compressed SMoE models to “imitate the outputs generated by the full SMoE model on the training dataset” (which is included in the eighth line of the last paragraph of Section 4.1 in our paper).

To further address your concerns, we conduct two additional experiments on COPA and MultiRC tasks: 

-   Knowledge distillation from full SMoE to dense model
-   Self-knowledge-distillation of full SMoE, i.e. at the iteration $t$, distilled from the full SMoE itself at the iteration $(t-1)$.

The results are shown in Table R5, comparing them with that of our dense and full-SMoE without KD. It demonstrates a clear performance decline when applying knowledge distillation to dense and full-SMoE models. 

Table R5: Comparison between {dense, distilled dense, full-SMoE, self-distilled full-SMoE} models on the COPA and MultiRC tasks.

| Model                    | COPA  | MultiRC |
| ------------------------ | ----- | ------- |
| Dense (Ours)             | 58.00 | 74.25   |
| Distilled dense          | 58.00 | 73.70   |
| Full-SMoE (Ours)         | 68.00 | 76.19   |
| Self-distilled Full-SMoE | 62.00 | 73.93   |

Reviewer comment (Experiments):
From Table 2, it seems performance of the model considerably drops after applying the compression technique (M-SMoE vs. MC-SMoE). The authors should provide more detailed analysis on the design of the compression method. For example, will different pruning strategies/schedules work better?

Author response:
**[Cons 7. Performance Drop Post-Compression]**

**[Motivation of the Post-merging Compression Design]** In our MC-SMoE, we approximate a weight matrix $\mathtt{W}$ using a combination of a low-rank matrix $\mathtt{U}\mathtt{V}$ and a sparse matrix $S$.

-   As depicted in Figure 4 of our paper, we note that M-SMoE encourages a reduced dimensionality in the weight space of merged experts. This reduction suggests that neurons within a weight matrix share a common subspace, akin to the coherent parts of these neurons. Consequently, low-rank matrices are well-suited for approximating these coherent parts effectively. 
-   A structured-pruned sparse matrix is frequently used for compressing $\mathtt{W}$ ([R14]). This process benefits from the separation of low-rank and sparse matrices, which facilitates easier pruning. The low-rank matrix effectively segregates the coherent and incoherent parts of neurons. By introducing a new matrix S, we can approximate the residual incoherent parts. This approach allows for the precise approximation of the remaining incoherent parts by adding a new matrix S, which is then pruned to remove the non-expressive, incoherent components.

**[Experiment - Comparison of Pruning Strategies]** We conduct extra comparison experiments with the following two other pruning strategies on the COPA and MultiRC tasks:

-   *Magnitude pruning*, proposed by [R14], preserves weights with high absolute values and is the most widely used method for weight pruning.
-   *Iterative pruning*, as introduced in [R15], targets the direct removal of neurons that have importance scores below a predefined threshold during each iteration.

The results are shown in Table R6, indicating that our pruning strategy in MC-SMoE demonstrates the best performance among the three strategies.

Table R6: Comparison between {magnitude pruning, iterative pruning, ours} pruning methods on the COPA and MultiRC tasks with the *switch-base-32* model.

| Method            | COPA    | MultiRC |
| ----------------- | ------- | ------- |
| Magnitude Pruning | $61.00$ | $73.51$ |
| Iterative Pruning | $66.00$ | $72.63$ |
| Ours              | $67.00$ | $73.98$ |

**[Experiment - Comparison of Different Schedules]** We conduct extra comparison experiments with two other pruning ratio schedules, i.e. the linear, quadratic pruning ratio schedules, and our cubic schedules, on the COPA and MultiRC tasks. The outcomes, shown in Table R7, reveal a performance hierarchy with our cubic method outperforming the quadratic and linear approaches.

Table R7: Comparison between different schedules on the COPA and MultiRC tasks with the *switch-base-32* model.

| Schedule     | COPA    | MultiRC |
| ------------ | ------- | ------- |
| Linear       | $59.00$ | $73.83$ |
| Quadratic    | $61.00$ | $73.92$ |
| Cubic (ours) | $67.00$ | $73.98$ |

Reviewer comment (Experiments):
The authors mention “further memory and parameter efficiency” in the paragraph above Algorithm 1. However, no experiments are conducted to evaluate the speed and memory of the MC-SMoE models. The latency results in Table A10 indicate that there is only marginal speed gain of M-SMoE compared with full-SMoE. The authors should benchmark the inference speed (throughput) and memory usage of M-SMoE and MC-MoE, and compare the metrics with the dense and the full MoE models.

Author response:
**[Cons 8. Evaluation of Speed and Memory]**

We appreciate the constructive suggestion of evaluating “the inference speed (throughput) and memory usage” of the models. We politely address your concerns from several aspects:

-   *<Expert merging improves “memory efficiency”.>* Due to the sparse activation characteristic of SMoE models, where each SMoE layer chooses one or two experts for the computation of each token, merging experts leads to a reduction in memory usage but not in computational cost (specifically FLOPs). In Table 2 of our paper, we showcase the model size and inference FLOPs for M-SMoE, MC-SMoE, and all other baseline models, illustrating the improvement in memory efficiency brought by both M-SMoE and MC-SMoE.
-   *<Theoretical speedup of expert merging exists.>* This is because, in conventional SMoE implementation, the routing process involves (1) a layout transform of the tensors (to arrange tokens targeting the same experts into a continuous memory buffer) and its reverse operation ([R13]), (2) breaking down one large matrix block GEMM operation into many smaller matrix block GEMM operations (each corresponding to an individual expert), leading to less efficient utilization of modern computational hardware's advantages. These factors lead to a decrease in real throughput for the sparsely activated computation in SMoE when the number of experts rises, a topic that remains an open area for research ([R13]) and is earmarked for exploration in our future studies. In other words, reducing the number of experts can yield improvements in actual inference speed, even while maintaining roughly constant FLOPs.
-   *<Why there is a “marginal speed (latency) gain of M-SMoE”?>* It is mainly because of the challenge of trimming the router's output channels without altering its routing functionality. Therefore, we execute the merging of experts by substituting all the expert weights in a group with a single, merged expert weight. This throughout analysis is included in the eighth line of Appendix 1.2 in our paper. 

[R14] https://arxiv.org/pdf/1506.02626.pdf

[R15] https://arxiv.org/pdf/1906.10771.pdf