From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Potential of memory-efficient MiniMoE
comment: Many thanks for the recognition of our contribution of lifting the curse.

There is no free lunch to improve the capacity of the student. We should argue that, in order to increase the capacity, memory/space consumption is a cheaper choice (e.g., more experts) than latency/time consumption (e.g., more operations), and this is potentially the reason why LLMs like [PaLM](https://arxiv.org/abs/2204.02311) and [FLAN](https://huggingface.co/google/flan-t5-xxl) could become so popular.

We should also highlight that scenarios that require rather limited memory consumption (e.g., on-device scenario) is currently not (though can be in the near future) the main concern of LLMs. In contrast, LLMs are usually served in GPU scenario, where memory/space is easy to access. And we have to admit the increased parameter amount is a critical limitation of MiniMoE, and we have discussed this in the revised paper (appendix G).

For the decomposition, the decomposition of embeddings is just an example. In fact, the parameters in transformers can also be decomposed. Specifically, we argue that, without much performance sacrifice, parameters in MiniMoE can be easier to decompose in comparison with MiniLM owing to the sparse activation property. That is, parameters in MiniMoE have lower rank than those in MiniLM, and this can be shown by analyzing the normalized singular values using SVD. The preliminary results of the output matrices of the last FFN layers separately from MiniLM_3L;384H and MiniMoE_3L;384H are shown in the following:

|Method|% Singular values >= 0.2|% Singular values >= 0.1|% Singular values >= 0.05|Est Trm Params|
|--|--|--|--|--|
|MiniLM_3L;384H|315/384=82%|356/384=93%|373/384=97%|5.3M/3+5.3M/3x2x93%=1.8M+3.3MM=5.1M|
|MiniMoE_3L;384H expert #1|6/384=2%|82/384=21%|275/384=72%|-|
|MiniMoE_3L;384H expert #2|34/384=9%|220/384=57%|361/384=94%|-|
|MiniMoE_3L;384H expert #3|15/384=4%|175/384=46%|338/384=88%|-|
|MiniMoE_3L;384H expert #4|24/384=6%|200/384=52%|357/384=93%|-|
|MiniMoE_3L;384H all experts|79/384/4=5%|677/384/4=44%|1331/384/4=87%|16.4M/9+16.4M/9x8x44%=1.8M+6.4M=8.2M|

With this finding, MiniMoE can compress more parameters than MiniLM does and finally yield a comparable parameter scale with that of MiniLM.

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_2V6h
title: Response
comment: Thanks for the answer, but I would argue to maintain my score. Below is my response:

I don't think the count of parameters after SVD matters, since it is orthogonal to your efforts and you can apply decomposition to the baseline methods as well. As shown in your table, MiniMoE uses 3x as many parameters in transformer attention matrices compared to MiniLM. It can be costly, especially when considering the situations of the application of model distillation. 

And yes, I agree that the main contribution of this paper is valid. MinoMoE is an effective way to lift the curse of large LMs. The only concern is if the cost is too high when the performance gain is limited.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer 89th (2/2)
comment: 4. **Implementation of MiniMoE.**

    We strictly follow the design of SwitchTransformer and extend it to the design of our MiniMoE. We also follow their associated appendices to implement an MoE for multihead attention. In detail, based on the original design, we treat an FFN/MHA block as an minimal expert, adopt top-1 gating with load balancing, and employ a capacity factor of 1.25 for a good tradeoff (where overflowed tokens are dropped). 
    
    For the parameter effect of adding an expert, take expanding MiniLM_4L;192H (11.33M) to MiniMoE_4L;192H_2E (14.91M) as an example, the number of parameters for embeddings is not changed (5.96M->5.96M), but adding an expert (1E->2E) results in an increased number of parameters for transformers (5.37M->8.96M). Here, we have to highlight that, while MoE can bring much extra memory consumption for LLMs like SwitchTransformer, it will not be that much for MiniMoE.

    Further, our design for HashLayer also strictly follows the original random hash design, i.e., per-token hash is used. For all cases, students are always randomly initialized following MiniLM.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer 89th (1/2)
comment: 1. **Evaluation on larger LMs and other benchmarks.**

    LM distillation, under either task-agnostic settings as in our paper or task-specific settings, has seldomly investigated to distil LMs larger than BERT_large. Even worser, there is only little work has been investigated to distil BERT_large under task-agnostic settings. A detailed list of language models used in previous studies is offered in the following table.
    
    |Method|Task-agnostic?|BERT_base|BERT_large|Larger LMs?|
    |--|--|--|--|--|
    |[PKD](https://arxiv.org/abs/1908.09355)|N|Y|N|N|
    |[DynaBERT](https://arxiv.org/abs/2004.04037)|N|Y|N|N|
    |[MoEBERT](https://arxiv.org/abs/2204.07675)|N|Y|N|N|
    |[AutoDisc](http://arxiv.org/abs/2205.14570)|N|Y|N|EncT5_1b|
    |[TinyBERT](https://arxiv.org/abs/1909.10351)|Y|Y|N|N|
    |[MiniLM](https://arxiv.org/abs/2002.10957)|Y|Y|N|N|
    |[MiniLMv2](https://arxiv.org/abs/2012.15828)|Y|Y|Y|N|
    |[DistilGPT2](https://huggingface.co/distilgpt2)|Y|Y*|N|N|
    |MiniMoE|Y|Y|Y|N|
    \* GPT2_base is at a similar scale to BERT_base.

    In our paper, we just follow their paces in task-agnostic settings, not only due to the huge scales of larger LMs like T5 and GPT-3 but also due to that LM distillation under task-agnostic settings requires the access to the original pretraining data of usually vast volume. Moreover, we have to argue that BERT_large is essentially a LLM in comparison with conventional LMs such as n-gram LMs. What's more, larger LMs like T5 can be incomparable to BERT_base and BERT_like owing to the architectural difference, where existing task-agnostic methods including ours may easily fail.

    In order to not confuse readers, we will rephrase all statements involving **large** language models. Accordingly, we change our title to *LIFTING THE CURSE OF CAPACITY GAP IN DISTILLING LANGUAGE MODELS*.

    Plus to that, GLUE is the most frequently-used benchmark for LLM distillation, so we choose it for comparable results. In addition to that, we also examine MiniMoE on CoNLL.

    Regarding all the considerations mentioned above, we try to check the existence of the curse of capacity gap and examine MiniMoE on a larger-scale setting, i.e., [BERT_base_Chinese](https://huggingface.co/bert-base-chinese) v.s. [BERT_1b_Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-MegatronBert-1.3B) on some datasets (TNews: topic classification; AFQMC: similar question matching; OCNLI: natural language inference) from [CLUE](https://www.cluebenchmarks.com/index.html) (can be viewed as the Chinese GLUE). The preliminary results are shown in the following. As far as we know, while BERT_1b in English trained by [Nvidia Megatron](https://github.com/NVIDIA/Megatron-LM#downloading-checkpoints) is not publicly available, BERT_1b_Chinese can be easily downloaded through huggingface. It is noteworthy that BERT_base_Chinese is trained on Chinese Wikipedia (\~15G) while BERT_1b_Chinese is trained on [Wudao Corpus](https://resource.wudaoai.cn/home?ind=1&name=WuDaoCorpora%202.0&id=1394901288847716352) (\~300G). We use Wikipedia data as the default choice for distillation, but Wudao data seems to be a more suitable (though not that fair) one for distilling BERT_1b_Chinese as indicated in the results. Painfully, it consumes around one week to achieve one epoch of distilling BERT_1b_Chinese using Wudao in contrast to five epochs of distilling BERT_base_Chinese/BERT_1b_Chinese on Wikipedia in one day. The results show that BERT_1b fails to realize better students than BERT_base does, and MiniMoE has the ability to solve the issue on the larger-scale setting.

    |Method|TNews|AFQMC|OCNLI|Avg.|
    |--|--|--|--|--|
    |base|
    |BERT_base_Chinese|57.0|74.8|75.4|69.1|
    |MiniLM_6L;384|55.5|72.0|71.0|66.2|
    |MiniMoE_6L;384|55.9|72.9|70.8|66.5|
    |1b|
    |BERT_1b_Chinese|60.0|76.1|79.2|71.7|
    |MiniLM_6L;384|48.2|69.4|63.0|60.2|
    |MiniMoE_6L;384|not done due the failure of MiniLM||||
    |MiniLM_6L;384*|54.9|70.7|69.9|65.2|
    |MiniMoE_6L;384*|56.7|72.4|71.0|66.7|
    \* Wudao for distillation with one epoch. 

2. **Practical inference compute.**

    Practical inference is partly offered in Table 5, in which communication cost is also considered in throughput. For the memory consumption, we have to admit that we mistakenly use inference compute as a mention only for inference latency, and shall revise this accordingly in the revised paper.

3. **Performance improvement of MiniMoE.**

    We conduct necessary t-tests for results in Table 3 and 4 where p-values are listed in the following table, which show that the performance differences on GLUE and CoNLL are significant.

    |Comparison Pair|p-value|
    |--|--|
    |MiniLM_4L;384H from BERT_base v.s. MiniMoE_4L;384H from BERT_base|0.0484|
    |MiniMoE_4L;384H from BERT_base v.s. MiniMoE_4L;384H from BERT_large|0.0479|

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer 2V6h
comment: 1. **MiniMoE is limited to transformer architecture.**

    MiniMoE can actually be easily extended to other architectures containing FFN-like sub-architectures.

2. **Training-efficiency and parameter concern for practical usage. The effect of number of experts to the total parameter amount.**

    While more training memory has been consumed by MiniMoE, MiniMoE does not notably slows down the training process since the latency is nearly intact. Both MiniLM and MiniMoE take about one day to converge using eight GPUs.
    
    While the parameter amount of MiniMoE is increased when compared with same-compute MiniLM, the increased parameter amount still lies in an acceptable region. At the same time, we could use embedding decomposition (e.g., SVD) as a way to reduce the total parameter amount since the number of embedding parameters takes a critical portion. That is, original embedding parameter amount is 30522x384, after decomposition it can become 30522x100+100x384 << 30522x384.

    |Method|Params|Trm. Params|Emb. Params|
    |--|--|--|--|
    |MiniLM_3L;384H|17.2M|5.3M|11.9M|
    |MiniMoE_3L;384H|28.3M|16.4M|11.9M|
    |MiniMoE_3L;384H w/ embedding SVD|19.4M|16.4M|3.0M|

    Further, we could continuously distil MiniMoE_3L;384H into MiniLM_3L;384H, and the results in the following table show that this only results in an acceptable performance degradation on large dataset like MNLI and SST-2 but undesired performance degradation on small datasets like RTE.

    |Method|MNLI-m/mm|SST-2|RTE|
    |--|--|--|--|
    |MiniLM_3L;384H|77.8/78.4|89.1|66.1|
    |MiniMoE_3L;384H|78.2/78.7|89.3|67.0|
    |MiniLM_3L;384H from MiniMoE_3L;384H|78.1/78.4|89.5|64.3|

    For the parameter effect of adding an expert, take expanding MiniLM_4L;192H (11.33M) to MiniMoE_4L;192H_2E (14.91M) as an example, the number of parameters for embeddings is not changed (5.96M->5.96M), but adding an expert (1E->2E) results in an increased number of parameters for transformers (5.37M->8.96M). Here, we have to again highlight that, while MoE can bring much extra memory consumption for LLMs like SwitchTransformer, it will not be that much for MiniMoE.

3. **Performance of MiniMoE.**

    We conduct necessary t-tests for results in Table 3 and 4 where p-values are listed in the following table, which show that the performance differences on GLUE and CoNLL are significant.

    |Comparison Pair|p-value|
    |--|--|
    |MiniLM_4L;384H from BERT_base v.s. MiniMoE_4L;384H from BERT_base|0.0484|
    |MiniMoE_4L;384H from BERT_base v.s. MiniMoE_4L;384H from BERT_large|0.0479|

    Plus to that, the aim of this paper to amplify the pain brought by the curse of capacity gap and put forward MiniMoE as the initial solution that can lift the curse, rather than a state of the art. The abilities of some previous studies to lift the curse are listed in the following table as a supplement to Table 1 in our paper.

    |Method|BERT_base|BERT_large|$\triangle$|Lift?|
    |--|--|--|--|--|
    |Teacher|86.7|88.3|+1.6|-|
    |DynaBERT_15%/5%*|81.1|79.2|-1.9|N|
    |KD_10%/5%*|81.3|80.8|-0.5|N|
    |AutoDisc_10%/5%*|82.4|82.1|-0.3|N|
    |TinyBERT_4L;312H|80.7|80.5|-0.2|N|
    |MiniLM_3L;384H|82.5|82.0|-0.5|N|
    |MiniLM_3L;384H w/ TA|82.3|82.2|-0.1|N|
    |MiniMoE_3L;384H|82.6|83.1|+0.5|Y|
    \* To ensure students at similar scales, the student/teacher ratios should be properly reduced.

    We hereby argue that the performance of MiniMoE is acceptable.

4. **Implementation of DeKD.**
    
    We strictly follow the best configuration of DeKD as reported in their paper, where alpha is 1.0 and beta is 8.0.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer dnTB (2/2)
comment: 3. **Better to use Figure 4 instead of Table 4, and show that other baselines fail to lift the curse.**

    We will update the presentation of Table 4 in the revised paper.

    We show that other baselines fail to life the curse in the following table.

    |Method|BERT_base|BERT_large|$\triangle$|Lift?|
    |--|--|--|--|--|
    |Teacher|86.7|88.3|+1.6|-|
    |DynaBERT_15%/5%*|81.1|79.2|-1.9|N|
    |KD_10%/5%*|81.3|80.8|-0.5|N|
    |AutoDisc_10%/5%*|82.4|82.1|-0.3|N|
    |TinyBERT_4L;312H|80.7|80.5|-0.2|N|
    |MiniLM_3L;384H|82.5|82.0|-0.5|N|
    |MiniLM_3L;384H w/ TA|82.3|82.2|-0.1|N|
    |MiniMoE_3L;384H|82.6|83.1|+0.5|Y|
    \* To ensure students at similar scales, the student/teacher ratios should be properly reduced.

4. **Initialization of MoE students.**

    For all cases, students are always randomly initialized following MiniLM, in which random initialization shows superiority to trivially inheritance from the teacher.

5. **Task-agnostic or task-specific results.**

    All the results are task-agnostic ones, except those in Table 6. We have explicitly denoted it in the caption in the revised paper.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer dnTB (1/2)
comment: 1. **Evaluation on larger LMs.**

    LM distillation, under either task-agnostic settings as in our paper or task-specific settings, has seldomly been investigated to distil LMs larger than BERT_large. Even worser, there is only little work has been investigated to distil BERT_large under task-agnostic settings. A detailed list of language models used in previous studies is offered in the following table.
    
    |Method|Task-agnostic?|BERT_base|BERT_large|Larger LMs?|
    |--|--|--|--|--|
    |[PKD](https://arxiv.org/abs/1908.09355)|N|Y|N|N|
    |[DynaBERT](https://arxiv.org/abs/2004.04037)|N|Y|N|N|
    |[MoEBERT](https://arxiv.org/abs/2204.07675)|N|Y|N|N|
    |[AutoDisc](http://arxiv.org/abs/2205.14570)|N|Y|N|EncT5_1b|
    |[TinyBERT](https://arxiv.org/abs/1909.10351)|Y|Y|N|N|
    |[MiniLM](https://arxiv.org/abs/2002.10957)|Y|Y|N|N|
    |[MiniLMv2](https://arxiv.org/abs/2012.15828)|Y|Y|Y|N|
    |[DistilGPT2](https://huggingface.co/distilgpt2)|Y|Y*|N|N|
    |MiniMoE|Y|Y|Y|N|
    \* GPT2_base is at a similar scale to BERT_base.

    In our paper, we just follow the paces in task-agnostic settings, not only due to the huge scales of larger LMs like T5 and GPT-3 but also due to that LM distillation under task-agnostic settings requires the access to the original pretraining data of usually vast volume. What's more, larger LMs like T5 can be incomparable to BERT_base and BERT_like owing to the architectural difference, where existing task-agnostic methods including ours may easily fail.

    We hereby argue that distilling LLMs like BERT is just a following pace due to the unaffordable expense of distilling LLMs like T5 or GPT-3 for us. In order to not confuse readers, we will rephrase all statements involving **large** language models. Accordingly, we change our title to *LIFTING THE CURSE OF CAPACITY GAP IN DISTILLING LANGUAGE MODELS*.

    Regarding all the considerations mentioned above, we try to check the existence of the curse of capacity gap and examine MiniMoE on a larger-scale setting, i.e., [BERT_base_Chinese](https://huggingface.co/bert-base-chinese) v.s. [BERT_1b_Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-MegatronBert-1.3B) on some datasets (TNews: topic classification; AFQMC: similar question matching; OCNLI: natural language inference) from [CLUE](https://www.cluebenchmarks.com/index.html) (can be viewed as the Chinese GLUE). The preliminary results are shown in the following. As far as we know, while BERT_1b in English trained by [Nvidia Megatron](https://github.com/NVIDIA/Megatron-LM#downloading-checkpoints) is not publicly available, BERT_1b_Chinese can be easily downloaded through huggingface. It is noteworthy that BERT_base_Chinese is trained on Chinese Wikipedia (\~15G) while BERT_1b_Chinese is trained on [Wudao Corpus](https://resource.wudaoai.cn/home?ind=1&name=WuDaoCorpora%202.0&id=1394901288847716352) (\~300G). We use Wikipedia data as the default choice for distillation, but Wudao data seems to be a more suitable (though not that fair) one for distilling BERT_1b_Chinese as indicated in the results. Painfully, it consumes around one week to achieve one epoch of distilling BERT_1b_Chinese using Wudao in contrast to five epochs of distilling BERT_base_Chinese/BERT_1b_Chinese on Wikipedia in one day. The results show that BERT_1b fails to realize better students than BERT_base does, and MiniMoE has the ability to solve the issue on the larger-scale setting.

    |Method|TNews|AFQMC|OCNLI|Avg.|
    |--|--|--|--|--|
    |base|
    |BERT_base_Chinese|57.0|74.8|75.4|69.1|
    |MiniLM_6L;384|55.5|72.0|71.0|66.2|
    |MiniMoE_6L;384|55.9|72.9|70.8|66.5|
    |1b|
    |BERT_1b_Chinese|60.0|76.1|79.2|71.7|
    |MiniLM_6L;384|48.2|69.4|63.0|60.2|
    |MiniMoE_6L;384|not done due the the failure of MiniLM||||
    |MiniLM_6L;384*|54.9|70.7|69.9|65.2|
    |MiniMoE_6L;384*|56.7|72.4|71.0|66.7|
    \* Wudao for distillation with one epoch. 

2. **Number of parameters as an indicator of parameter capacity instead of GFLOPs.**

    Besides Table 5 in the paper, we here additional report the number of parameters for methods in Table 3 in the following table, which in fact shows that the number of parameters is positively correlated to GFLOPs.

    |Method|GFLOPs|Params|Trm. Params|Emb. Params|
    |--|--|--|--|--|
    |KD_5%|0.54M|28.7M|4.9M|23.8M|
    |AutoDisc_5%|0.54M|28.7M|4.9M|23.8M|
    |TinyBERT_4L;312H|0.60|14.5M|4.8M|9.7M|
    |MiniLM_3L;384H|0.68|17.2M|5.3M|11.9M|
    |MiniMoE_3L;384H|0.68|28.3M|16.4M|11.9M|

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer yXUa (2/2)
comment: 3. **Tradeoff of using larger LMs as teachers in Figure 2.**

    Note that a vertical grid in the figure actually stands for 2 units or 1 unit. That is to say, although two lines in a figure seems to locate very close to each other, they are far apart from each other if we zoom in. However, if we zoom in, we will lose some data points.

4. **Performance difference of BERT_base v.s. BERT_large of MiniMoE is not as large as that of BERT_base v.s. BERT_large.**

    We conduct necessary t-tests for results in Table 4 where p-values are listed in the following table, which show that the performance differences on GLUE and CoNLL are significant.

    |Comparison Pair|p-value|
    |--|--|
    |MiniMoE_4L;384H from BERT_base v.s. MiniMoE_4L;384H from BERT_large|0.0479|

    While the absolute performance improvement is not large when compared with the absolute performance improvement of BERT_large v.s. BERT_base, it actually can be acceptable that reduced LMs produce reduced improvement.

5. **The use of softmax in top-1 gating.**

    We strictly follow the design of SwitchTransformer and extend it to the design of our MiniMoE.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer yXUa (1/2)
comment: 1. **Novelty of MiniMoE is in question.**

    We have to again highlight that the recognized curse of capacity gap can be a pain in distilling LLMs, and MiniMoE is initially a solution and currently the only solution to lift the curse. This can be a great breakthrough in the area and makes a significant impact. By the way, the abilities of some previous studies to lift the curse are listed in the following table as a supplement to Table 1 in our paper.

    |Method|BERT_base|BERT_large|$\triangle$|Lift?|
    |--|--|--|--|--|
    |Teacher|86.7|88.3|+1.6|-|
    |DynaBERT_15%/5%*|81.1|79.2|-1.9|N|
    |KD_10%/5%*|81.3|80.8|-0.5|N|
    |AutoDisc_10%/5%*|82.4|82.1|-0.3|N|
    |TinyBERT_4L;312H|80.7|80.5|-0.2|N|
    |MiniLM_3L;384H|82.5|82.0|-0.5|N|
    |MiniLM_3L;384H w/ TA|82.3|82.2|-0.1|N|
    |MiniMoE_3L;384H|82.6|83.1|+0.5|Y|
    \* To ensure students at similar scales, the student/teacher ratios should be properly reduced.
    
    We hereby argue that MiniMoE is simple but can be impactful.

2. **Evaluation on larger LMs.**

    LM distillation, under either task-agnostic settings as in our paper or task-specific settings, has seldom been investigated to distil LMs larger than BERT_large. Even worse, there is only little work has been investigated to distil BERT_large under task-agnostic settings. A detailed list of language models used in previous studies is offered in the following table.
    
    |Method|Task-agnostic?|BERT_base|BERT_large|Larger LMs?|
    |--|--|--|--|--|
    |[PKD](https://arxiv.org/abs/1908.09355)|N|Y|N|N|
    |[DynaBERT](https://arxiv.org/abs/2004.04037)|N|Y|N|N|
    |[MoEBERT](https://arxiv.org/abs/2204.07675)|N|Y|N|N|
    |[AutoDisc](http://arxiv.org/abs/2205.14570)|N|Y|N|EncT5_1b|
    |[TinyBERT](https://arxiv.org/abs/1909.10351)|Y|Y|N|N|
    |[MiniLM](https://arxiv.org/abs/2002.10957)|Y|Y|N|N|
    |[MiniLMv2](https://arxiv.org/abs/2012.15828)|Y|Y|Y|N|
    |[DistilGPT2](https://huggingface.co/distilgpt2)|Y|Y*|N|N|
    |MiniMoE|Y|Y|Y|N|
    \* GPT2_base is at a similar scale to BERT_base.

    In our paper, we just follow the paces in task-agnostic settings, not only due to the huge scales of larger LMs like T5 and GPT-3 but also due to that LM distillation under task-agnostic settings requires the access to the original pretraining data of usually vast volume. What's more, larger LMs like T5 can be incomparable to BERT_base and BERT_like owing to the architectural difference, where existing task-agnostic methods including ours may easily fail.

    We hereby argue that distilling LLMs like BERT is just a following pace due to the unaffordable expense of distilling LLMs like T5 or GPT-3 for us. In order to not confuse readers, we will rephrase all statements involving **large** language models. Accordingly, we change our title to *LIFTING THE CURSE OF CAPACITY GAP IN DISTILLING LANGUAGE MODELS*.

    Regarding all the considerations mentioned above, we try to check the existence of the curse of capacity gap and examine MiniMoE on a larger-scale setting, i.e., [BERT_base_Chinese](https://huggingface.co/bert-base-chinese) v.s. [BERT_1b_Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-MegatronBert-1.3B) on some datasets (TNews: topic classification; AFQMC: similar question matching; OCNLI: natural language inference) from [CLUE](https://www.cluebenchmarks.com/index.html) (can be viewed as the Chinese GLUE). The preliminary results are shown in the following. As far as we know, while BERT_1b in English trained by [Nvidia Megatron](https://github.com/NVIDIA/Megatron-LM#downloading-checkpoints) is not publicly available, BERT_1b_Chinese can be easily downloaded through huggingface. It is noteworthy that BERT_base_Chinese is trained on Chinese Wikipedia (\~15G) while BERT_1b_Chinese is trained on [Wudao Corpus](https://resource.wudaoai.cn/home?ind=1&name=WuDaoCorpora%202.0&id=1394901288847716352) (\~300G). We use Wikipedia data as the default choice for distillation, but Wudao data seems to be a more suitable (though not that fair) one for distilling BERT_1b_Chinese as indicated in the results. Painfully, it consumes around one week to achieve one epoch of distilling BERT_1b_Chinese using Wudao in contrast to five epochs of distilling BERT_base_Chinese/BERT_1b_Chinese on Wikipedia in one day. The results show that BERT_1b fails to realize better students than BERT_base does, and MiniMoE has the ability to solve the issue on the larger-scale setting.

    |Method|TNews|AFQMC|OCNLI|Avg.|
    |--|--|--|--|--|
    |base|
    |BERT_base_Chinese|57.0|74.8|75.4|69.1|
    |MiniLM_6L;384|55.5|72.0|71.0|66.2|
    |MiniMoE_6L;384|55.9|72.9|70.8|66.5|
    |1b|
    |BERT_1b_Chinese|60.0|76.1|79.2|71.7|
    |MiniLM_6L;384|48.2|69.4|63.0|60.2|
    |MiniMoE_6L;384|not done due the the failure of MiniLM||||
    |MiniLM_6L;384*|54.9|70.7|69.9|65.2|
    |MiniMoE_6L;384*|56.7|72.4|71.0|66.7|
    \* Wudao for distillation with one epoch.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer tQUP (2/2)
comment: 3. **A focused background section.**
    
    Sorry for the confusion. We have done that in the revised paper (appendix E).

4. **Number of parameters in addition to GFLOPs.**
    
    Besides Table 5 in the paper, we here additional report the number of parameters for methods in Table 3 in the following table, which in fact shows that 1) the number of parameters is positively correlated to GFLOPs and 2) the parameters is mainly composed of the parameters from embeddings.

    |Method|GFLOPs|Params|Trm. Params|Emb. Params|
    |--|--|--|--|--|
    |KD_5%|0.54M|28.7M|4.9M|23.8M|
    |AutoDisc_5%|0.54M|28.7M|4.9M|23.8M|
    |TinyBERT_4L;312H|0.60|14.5M|4.8M|9.7M|
    |MiniLM_3L;384H|0.68|17.2M|5.3M|11.9M|
    |MiniMoE_3L;384H|0.68|28.3M|16.4M|11.9M|

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: Response to reviewer tQUP (1/2)
comment: 1. **MiniMoE is not novel.**

    We have to again highlight that the recognized curse of capacity gap can be a pain in distilling LLMs, and MiniMoE is initially a solution and currently the only solution to lift the curse. This can be a great breakthrough in the area and makes a significant impact. The abilities of some previous studies to lift the curse are listed in the following table as a supplement to Table 1 in our paper.

    |Method|BERT_base|BERT_large|$\triangle$|Lift?|
    |--|--|--|--|--|
    |Teacher|86.7|88.3|+1.6|-|
    |DynaBERT_15%/5%*|81.1|79.2|-1.9|N|
    |KD_10%/5%*|81.3|80.8|-0.5|N|
    |AutoDisc_10%/5%*|82.4|82.1|-0.3|N|
    |TinyBERT_4L;312H|80.7|80.5|-0.2|N|
    |MiniLM_3L;384H|82.5|82.0|-0.5|N|
    |MiniLM_3L;384H w/ TA|82.3|82.2|-0.1|N|
    |MiniMoE_3L;384H|82.6|83.1|+0.5|Y|
    \* To ensure students at similar scales, the student/teacher ratios should be properly reduced.
    
    We hereby argue that MiniMoE is simple but can be impactful.

2. **Evaluation on more modern and larger LMs.**

    LM distillation, under either task-agnostic settings as in our paper or task-specific settings, has seldom been investigated to distil LMs larger than BERT_large. Even worse, there is only little work has been investigated to distil BERT_large under task-agnostic settings. A detailed list of language models used in previous studies is offered in the following table.
    
    |Method|Task-agnostic?|BERT_base|BERT_large|Larger LMs?|
    |--|--|--|--|--|
    |[PKD](https://arxiv.org/abs/1908.09355)|N|Y|N|N|
    |[DynaBERT](https://arxiv.org/abs/2004.04037)|N|Y|N|N|
    |[MoEBERT](https://arxiv.org/abs/2204.07675)|N|Y|N|N|
    |[AutoDisc](http://arxiv.org/abs/2205.14570)|N|Y|N|EncT5_1b|
    |[TinyBERT](https://arxiv.org/abs/1909.10351)|Y|Y|N|N|
    |[MiniLM](https://arxiv.org/abs/2002.10957)|Y|Y|N|N|
    |[MiniLMv2](https://arxiv.org/abs/2012.15828)|Y|Y|Y|N|
    |[DistilGPT2](https://huggingface.co/distilgpt2)|Y|Y*|N|N|
    |MiniMoE|Y|Y|Y|N|
    \* GPT2_base is at a similar scale to BERT_base.

    In our paper, we just follow the paces in task-agnostic settings, not only due to the huge scales of larger LMs like T5 and GPT-3 but also due to that LM distillation under task-agnostic settings requires the access to the original pretraining data of usually vast volume. What's more, larger LMs like T5 can be incomparable to BERT_base and BERT_like owing to the architectural difference, where existing task-agnostic methods including ours may easily fail.

    We hereby argue that distilling LLMs like BERT is just a following pace due to the unaffordable expense of distilling LLMs like T5 or GPT-3 for us. In order to not confuse readers, we will rephrase all statements involving **large** language models. Accordingly, we change our title to *LIFTING THE CURSE OF CAPACITY GAP IN DISTILLING LANGUAGE MODELS*.
    
    Regarding all the considerations mentioned above, we try to check the existence of the curse of capacity gap and examine MiniMoE on a larger-scale setting, i.e., [BERT_base_Chinese](https://huggingface.co/bert-base-chinese) v.s. [BERT_1b_Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-MegatronBert-1.3B) on some datasets (TNews: topic classification; AFQMC: similar question matching; OCNLI: natural language inference) from [CLUE](https://www.cluebenchmarks.com/index.html) (can be viewed as the Chinese GLUE). The preliminary results are shown in the following. As far as we know, while BERT_1b in English trained by [Nvidia Megatron](https://github.com/NVIDIA/Megatron-LM#downloading-checkpoints) is not publicly available, BERT_1b_Chinese can be easily downloaded through huggingface. It is noteworthy that BERT_base_Chinese is trained on Chinese Wikipedia (\~15G) while BERT_1b_Chinese is trained on [Wudao Corpus](https://resource.wudaoai.cn/home?ind=1&name=WuDaoCorpora%202.0&id=1394901288847716352) (\~300G). We use Wikipedia data as the default choice for distillation, but Wudao data seems to be a more suitable (though not that fair) one for distilling BERT_1b_Chinese as indicated in the results. Painfully, it consumes around one week to achieve one epoch of distilling BERT_1b_Chinese using Wudao in contrast to five epochs of distilling BERT_base_Chinese/BERT_1b_Chinese on Wikipedia in one day. The results show that BERT_1b fails to realize better students than BERT_base does, and MiniMoE has the ability to solve the issue on the larger-scale setting.

    |Method|TNews|AFQMC|OCNLI|Avg.|
    |--|--|--|--|--|
    |base|
    |BERT_base_Chinese|57.0|74.8|75.4|69.1|
    |MiniLM_6L;384|55.5|72.0|71.0|66.2|
    |MiniMoE_6L;384|55.9|72.9|70.8|66.5|
    |1b|
    |BERT_1b_Chinese|60.0|76.1|79.2|71.7|
    |MiniLM_6L;384|48.2|69.4|63.0|60.2|
    |MiniMoE_6L;384|not done due the the failure of MiniLM||||
    |MiniLM_6L;384*|54.9|70.7|69.9|65.2|
    |MiniMoE_6L;384*|56.7|72.4|71.0|66.7|
    \* Wudao for distillation with one epoch.

From: ICLR.cc/2023/Conference/Paper1588/Authors
title: General response
comment: We would like to thank all the reviewers for their constructive comments. 
Specifically, we appreciate that 1) reviewers tQUP, dnTB, 2V6h, 89th agree that the curse of capacity is an **important** problem to study, e.g., reviewer tQUP points out that `curse of capacity gap is definitely an important thing to understand/address`; 2) reviewers tQUP, dnTB, 89th agree that MiniMoE is **simple yet effective**, e.g., reviewer tQUP points out that `MiniMoE method is simple and attains strong performance relative to baselines`; 3) reviewer yXUa admits that MiniMoE shows encouraging **empirical** results in `Empirical results look encouraging when we have limited compute`; 4) reviewer dnTB admits that the capacity gap is **theoretically** connected to VC dimension in `The problem is well posed and diagnosed with some enjoyable theoretical connections to VC dimension`.

The main concern from the reviewers is the **novelty** of MiniMoE. However, we should argue that the novelty of MiniMoE does not lie in how fancy the methodology can be, but in: 1) the recognized cures of capacity gap is of great importance as agreed by reviewers tQUP, dnTB, 2V6h, 89th; 2) the proposed MiniMoE is simple but currently the only solution to the curse, which serves as a zero-to-one method in lifting the curse, as agreed by reviewers tQUP, dnTB, 89th; 3) our empirical and theoretical results are promising as admitted by reviewers yXUa and dnTB. We believe that this can be a great breakthrough in the area, or at least an appealing initial point, to lift the curse of capacity gap.

Despite the simplicity of MiniMoE, we should highlight that the novelty of a method not limited to the design complexity as supported by Occam's razor that `entities should not be multiplied unnecessarily`. If a simple solution like MiniMoE can work out effectively, we can certainly take it as an initial point to motivate succeeding work in the area.

Some questions that commonly raised from reviewers are listed in the following, which are then later discussed in detail in individual responses to reviewers.

1. The impact of MiniMoE.

    MiniMoE is the initial and the only solution to lift the curse of capacity gap in comparison with related studies.

2. Experiments on LMs like T5 or GPT-3.
    
    Most studies in LM distillation have only been investigated to distil BERT_base due to compute expense and methodological incompatibility. Distilling decoder-involved LLMs has different behavior, as shown in [an award-winning paper](https://arxiv.org/abs/2203.10705). In order to not confuse readers, we will rephrase all statements involving **large** language models. Accordingly, we will change our title to *LIFTING THE CURSE OF CAPACITY GAP IN DISTILLING LANGUAGE MODELS*. Plus to that, GLUE is the most frequently-used benchmark for LLM distillation, so we choose it for comparable results.

3. Performance of MiniMoE in Table 3 and 4 is significant. Lines in Figure 2 and 4 significantly differ from each other. 
    
    We conduct t-tests for results in Table 3 and 4. And we make a note on Figure 2 and 4.

4. Implementation of MiniMoE.
    
    We strictly follow the design of SwitchTransformer and extend it to the design of our MiniMoE.

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_tQUP
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents MiniMoe, a mixture of experts (MoE) method for distilling large language models. The paper motivates this work by discussing the “curse of capacity gap”, which is the empirical finding that for a student of a fixed size, it is often better to distill from a smaller teacher than from a larger (and better performing) teacher (for which there is too large a “capacity gap” between the student and this teacher). It shows that existing LLM distillation methods all seem to suffer from this curse, but MiniMoE does not (distilling from BERT large always performs at least as well as distilling form BERT-base, across a very wide range of compressing factors). Additionally, it demonstrates that across several computational budgets (measured in GFLOPS), MiniMoE attains the best GLUE performance, relative to baseline BERT distillation methods.

The MiniMoE method is relatively straightforward: Each FFN network in the transformer architecture is replaced with ‘m’ FFN networks, combined with a gating mechanism which determines, for each token, which of the ‘m’ FFN networks the token should be routed to. Thus, though the number of parameters (and thus, the capacity of the model) increases as ‘m’ increases, the inference compute time does not. The paper also allows configuring MiniMoE such that there are also multiple “experts” + gating for the multi-headed attention layers (in addition to the FFN layers).

strength_and_weaknesses: Strengths
- Distilling large language models is an important problem, and this “curse of capacity gap” is definitely an important thing to understand/address—ideally, we would want better teachers to always lead to better students.
- The MiniMoe method is simple and attains strong performance relative to baselines when distilling BERT (base/large).
- The empirical analysis, studying the performance of BERT distillation across a range of student sizes, and both base/large teacher architectures, is relatively comprehensive. The paper compares MiniMoE to a relatively large list of baselines.

Weaknesses
- Methodologically the method is not novel. Uses a standard mixture of expert approach on a standard BERT architecture. The results are not super surprising or striking.
- BERT is a method from 2018. It would be good to study more modern and larger models/architectures, and understand how the methods work in these settings (and how the “curse of capacity cap” manifests in these models).
- The presentation/clarity could be improved. In many places, the language is very conversational, and goes on tangents about related work in the middle of presenting the method. Would be better to be a bit more focused, keeping the background/related work discussion more contained to their own sections.
- It would be good to include the number of parameters, in addition to the GFLOPS for inference, for all of the models.

clarity,_quality,_novelty_and_reproducibility: - Clarity: Medium clarity. Presentation could be polished a bit further.
- Quality: High quality work.
- Novelty: Relatively low novelty, as discussed above.
- Reproducibility: No code is provided as far as I can tell, though I believe the method would be relatively straightforward to implement.

summary_of_the_review: The paper presents an effective method for distilling BERT language models, and shows it attains SOTA performance at high compression rates on GLUE, and does not exhibit the “curse of capacity gap”. However, the method is a standard application of a mixture of experts (MoE) architecture (limited novelty), and the experiments are all on BERT, leaving open the question of how this distillation method would work on larger language models. As a result, for now I am recommending a soft reject, given the limited novelty/relevance of the work.


correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_yXUa
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper focuses on a distillation scenario where the teacher and student have a large capacity gap. Authors claim, in such cases, any student model cannot improve their performance no matter what teacher model is used due to the limited capacity of the student model. In order to reduce the performance gap in this case, the paper proposed a MoE-based solution that can increase the model capacity with minimal impact to the student model inference cost.

strength_and_weaknesses: **Strength**
* Ablation studies in the paper do provide various perspectives on the experiments.
* Empirical results look encouraging when we have limited compute (however, it’s not too surprising because it’s just because sparse MoEs can be more compute-efficient).

**Weakness**
* Novelty is in question. Both the theory and the proposed algorithm are not innovative enough. Particularly the proposed methodology, MiniMoE is just a renaming of a standard sparse MoE on FFN. I do not see any difference between standard MoE approaches where they can also grow model capacity with the minimal impact to the inference cost. The way it’s trained is even exactly the same as (Wang et al., 2021b) as the authors pointed out.
* Unlike the pitch from the paper, empirical evaluations focus on relatively small models, BERT base and BERT large, which commonly are not called “LLMs”?
* Unlike the theoretical analysis (Theorem 1 or Remark1) Figure 2 does not clearly show there’s a trade-off of using the larger model. The two lines look close enough in the noise range.
* Even though authors claim utilizing MoE can close the gap significantly, Table 4 does not seem to differentiate between using BERT base teacher vs BERT large teacher even with the proposed method (c.f. BERT base and BERT large are significantly different in their down-stream performance e.g. GLUE).

clarity,_quality,_novelty_and_reproducibility: **Clarity**
* The writing of the paper can be improved, particularly Section 3. Section 3 starts with describing the motivation and then rushed to implementation. The section lacks a logical flow of the actual proposal. It’s just scattered all around Section 2 and 3.
* MiniMoE chose top-1 gating (noted in page 5). Why do we need softmax?

**Novelty**
* As discussed in the weakness section, the novelty of this paper is limited.

summary_of_the_review: This paper does not provide enough novelty in this community.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_dnTB
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper is interested in distilling large language models and more specifically, LLMs of the size of BERT-base (110M params) and BERT-large (340M params) and the capacity gap between teacher and student. The authors formalize “the curse of capacity gap” where a student model gets a worse performance when distilling from a bigger teacher as the capacity gap (i.e. difference in number of parameters) between the student and the teacher grows.
The main contribution of the paper is to propose a mixture of expert based students which allows giving a lot of parameter capacity to the student without increasing its inference time. The authors show that this allows lifting the curse of capacity gap.

strength_and_weaknesses: Strengths:
- The problem is well posed and diagnosed with some enjoyable theoretical connections to VC dimension.
- The proposed method seems relatively straightforward and relatively effective in lifting the curse of capacity gap.

Weaknesses:
- The study is done on a relatively small scale (BERT-large is the biggest teacher used), which naturally leads to asking how well such a method would hold on the large language models of today (i.e. at least 10B scale).
- The writing is often quite confusing and does not fully back the claims in the introduction. While the story of the paper is about capacity (which I interpreted as parameter capacity), the results are presented with gflops throughout the paper instead of parameter count. The main result of the paper about the curse of the capacity gap is relegated to analyses into a hard-to-parse table 4. Other less critical presentation suggestions later.

clarity,_quality,_novelty_and_reproducibility: Questions:
- I did not understand how you initialize the experts of the Moe student. Could you clarify that?
- It is not clear what results are task-agnostic distillation and which are task-specific distillation. I think it would be useful to make that distinction more prominent either in writing or in the figure/table's captions.
- The paper talks about capacity of the student model, but I could not find parameter counts in any of the presented results. How big of a student do you end up with? Essentially, it would be more informative to see Figure 2, Table 4, and Figure 4 with parameter counts instead of gflops.
- Alternatively, I would suggest explaining better the link between gflops and parameter capacity and how gflop is the right metric to look at when studying the problem of the curse of capacity gap. 
- Table 3 shows a lot of different baselines and how mini-moe outperform them for a given gflop count. I think one missing piece of the argument is to show that other methods fail to lift the curse of capacity gap. Concretely, it would be insightful to show Table 4/Figure 4 with the other methods.

Presentation suggestion:
- My understanding is that the main point of the paper is Table 4/Figure 4. This should be frontloaded earlier in the result section. Table 3 is great although it serves a different purpose (mini-moe outperforms previous methods for a given gflops count).
- Page 9 - Failure of vision model -> this is a promising section, although it’s too short and too imcomplete. I would suggest presenting more complete results or removing this paragraph.
- Table 4 is hard to parse and partially redundant with Figure 4. I have found figure 4 to be easier to read. I would suggest translating Table 4’s aggregated result into Figure 4.

summary_of_the_review: The presentation of the paper lead me to doubt some of the result and the claims formulated in the introduction (lifting the curse of the capacity gap). In particular, the parameter counts are not specified with makes me question the size of the student teachers.

I am recommending a reject for now, but I would willing to raise my score if my questions and comments are taken into account to improve the paper.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_2V6h
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper identifies an important problem in knowledge distillation on large language models: The big gap between the sizes of teacher and student models could lead to poor performance. The authors analyze this problem theoretically from the perspective of learning theory and verify their claims with experiments on language modeling.

To solve this problem for transformer-based language models, this paper proposes to set up a "minimal" mixture of experts (MiniMoE) on each layer of the transformers on multi-head self-attention and fully connected networks. During knowledge distillation, a distribution of experts is learned, while only the expert with the highest score will be used during inference. Therefore, the model takes the advantage of larger student models while also keeping minimal computation during inference. Experiments show their method can improve the performance of distillation with the same computation compression ratio.
strength_and_weaknesses: 1. The paper identifies a problem that is not well-addressed by the previous paper: The size gap between student and teacher models matters for performance. It explains why previous methods work for small models (e.g. BERT-base) but may fail for larger language models (e.g. BERT-large). The claim is analyzed both theoretically and empirically.
2. The method proposed paves a new path towards size mismatching: Train a larger student network with multiple "minimal" experts during training while letting the model use only one expert during inference.

However, the paper is limited for the following reasons:

1. The model is designed specifically for transformer-based language models. As the formulation of minimal experts is implanted into modules of transformers, this method might not be applied to other architectures.
2. Though MiniMoE does not introduce more computation because of the number of experts, it does not save the training computation or the number of parameters. With 4 or 8 experts for MHA and FFN, the model size can be much larger than a single model. It can greatly limit the practical usage of distilled models.
3. Given the amount of parameters, MiniMoE does not show a great advantage over other KD methods. In table 4, MiniMoE shows observable improvement over its baseline only with the highest compression ratio. 

Questions:

1. Can you briefly discuss how the number of experts affects the size of parameters?
2. At the end of section 4, you mentioned that DeKD failed to accomplish the NLP tasks. Can you give more details about the experiments, e.g. how do you configure the alpha and beta?
clarity,_quality,_novelty_and_reproducibility: The paper is well written, with a clear description of its methods. The method is novel in that MiniMoE can take advantage of its model size while keeping minimal computation costs. Model and training details in the appendices make the paper easy to reproduce.
summary_of_the_review: The paper proposes a method to improve the efficiency of knowledge distillation when the student model is much smaller than the teacher. The method is novel, but it is limited in its parameter size and cannot be easily extended to architectures other than transformers.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1588/Reviewer_89th
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper aims to tackle the problem of "curse of capacity cap" for distillation of LLM. The authors provided a brief theoretical analysis of the problem, and proposed a solution of using MiniMoE as the student model architecture. Evaluations were conducted on two benchmarks (GLUE and CoNLL), where the authors compared to a standard dense student model and showed improvement in accuracy. Further analysis were done to study several design choices such as routing method, the number of experts, etc.
strength_and_weaknesses: Strengths:
* This paper investigates an important problem as the state-of-the-art LLM is becoming larger and larger and therefore distillation performance may drop as the capacity gap between teacher and student models enlarges.
* The proposed approach is a promising direction with several potential efficiency benefits from MoEs. 

Weaknesses:
- The experiments conducted in this paper is not very representative of the state-of-the-art LLM, which as a result makes the insight from this paper very limited. Specifically, a major class of LLMs are causal LM (decoder-only, e.g. GPT-3) and encoder-decoder (e.g. T5) models at the scale of >> 1B parameters, while this paper only experiments with BERT-large. 
- The authors state that the motivation and advantage of MiniMoE is that it "introduces almost no additional inference compute". This claim is vague and does not convey the whole picture. Specifically, "inference compute" should discuss both inference latency and memory consumption of the model, while the discussion on the latter is lacking. Another weakness is that even for the former the paper only provided theoretical FLOPs in major experiments while empirical metrics should be provided as well. 
clarity,_quality,_novelty_and_reproducibility: Quality:
- Experiments coverage is limited with only BERT models and two benchmarks, making the results less relevant for major LLMs (e.g. GPT-3, T5, etc) and benchmarks (e.g. SuperGLUE, few-shot in-context learning).
- Lots of the results are very close to baseline and it's unclear whether those improvement are statistically significant or not.
- Empirical throughput of MoEs depends on all-to-all communication, which is not mentioned in this paper. 

Clarity:
- This paper also experimented with MoE for MHA, which involves extra complexity than MoE for FFN only. But the implementation details are not provided.
- What's the expert capacity factor at inference time? What happens to tokens exceeding the expert capacity?
- This paper compared HashLayer as an alternative routing method, but the implementation was not discussed, such as was the hashing based on single token?
- Regarding the study on the number of experts, were the total number of the parameters held constant?

Important implementation details such as above are missing or not clear, which renders the paper at the risk of not reproducible. 
summary_of_the_review: This paper studies an important problem and explored a promising direction. However, the current experiments are limited in terms of understanding the effectiveness of the proposed solution for state-of-the-art LLMs. Another concern is the lack of evaluation/discussion of empirical performance of the proposed method in both inference and training time. Furthermore, the paper could be strengthened with more clarity on several implementation details.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough