From: ICLR.cc/2023/Conference/Paper1589/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1589/Authors
title: Response to reviewer PgbV
comment: 1. **Definition of *scale*.**

    We make it clear that the *scale* stands for the number of layers in Figure 1 and the student/teacher parameter scale ratio in Figure 3. We update Figure 1 in the revised paper.

2. **Sandwich-optimization is unclear.**

    To make the algorithm for once-for-all optimization clearer, we will update the algorithmic flow of once-for-all optimization in the revised paper. We currently present it with the following pseudo-code in a pytorch-like style. Note that here we use sparsity which equals to 1 - student/teacher parameter scale ratio in the paper. We certainly plan to release our code in the near future.

    ```python
        # given teacher t_model, train dataset train_data, evaluation dataset eval_data, initialize ta_model with copy().
        ta_model = copy(t_model)

        # specification.
        sparsities = [for x in range(0.05, 1.1, 0.05)]
        # subsample sparsities with eta.
        sparsities = subsample(sparsities, eta)
        sparsity_map = {}
        for s in sparsities:
            # obtain and store which parameters should be pruned for a given sparsity, with prune().
            sparsity_map[s] = prune(t_model, s)

        # optimization.
        for batch in train_data:
            t_output = t_model(batch)
            for s in sparsities:
                # obtain a candidate in-place with sparsify().
                sparsify(ta_model, s)
                ta_output = ta_model(batch)
                # compute loss and backward the loss.
                loss = loss_fn(t_output, ta_output)
                loss.backward()
            # update.
            optimizer.step()
            optimizer.zero_grad()

        # selection.
        # evaluate and store normalized performance of each with evaluate().
        performance = {}
        for s in sparsities:
            sparsify(ta_model, s)
            performance[s] = evaluate(ta_model, eval_data)
        tradeoff = {}
        for s in sparsities:
            tradeoff[s] = performance[s] + lam * s
        best_sparsity = sorted(tradeoff.items(), key=lambda x:x[1], reverse=True)[0]
        best_ta_model = sparsify(ta_model, best_sparsity)
    ```
    
    To make it clearer, these parameter-shared TAs can be optimized with a optimization loop.

3. **Selection is akin to a linear programming.**
    
    Specification determines scales of TAs and optimization determines performance. These scales and performance form feasible regions, linear programming then refers to finding the maximum tradeoff value during selection.

4. **Results in Figure 1.**

    We will update Figure 1 in the revised paper where a pruned student (90% sparsity) is used and the results on MRPC and QQP are reported. 

5. **Results of $\lambda$.**

    We add results on more $\lambda$ as shown in the following table. We observe that the choice of $\lambda$ is stable in a wide range. However, we also find that the performance can saturate since larger $\lambda$ favors larger sparsity and the effect of even larger $\lambda$ can be waived if the sparsity can not be further increased (e.g., a 15% TA is the smallest TA for a 10% student).

    |$\lambda$|MRPC|QQP|
    |--|--|--|
    |BERT_10%|87.8|84.6|
    |ManDisc_10%|88.2|84.8|
    |0.1|87.5|85.2|
    |0.2|88.4|84.9|
    |0.3|87.5|84.7|
    |0.5|87.8|84.7|
    |0.7|87.8|84.7|

6. **Pruning can lead to better performance in NLP.**
    
    While super tickets can improve performance on tasks, it can only achieve improvements in low sparsity levels. Thus, it is not controversial since slightly-pruned TAs can perform better but not essentially bring better student due to the capacity gap. For Figure 1 right, the students get better results simply due to reduced capacity gap.

From: ICLR.cc/2023/Conference/Paper1589/Authors
title: Response to reviewer LkAS
comment: 1. **Will the students supervised by a smaller TA consistently outperform the students supervised by a larger one?**
    
    We state that, if their performances exhibit no big differences, then the answer to this question is yes with a high probability. Sometimes, even smaller TAs with slightly lower performance can lead to better students according to our preliminary studies. This phenomenon is not quite new as supported by previous literature in the related work section in the paper. 

2. **Necessary ablation studies.**
    
    The ablation studies can actually be viewed as a process of bridging mandisc to autodisc by firstly adding $\lambda$-tradeoff, then adding once-for-all optimization. We present the results in the following table. The results show that: 1) (ManDisc v.s. ManDisc w/ $\lambda$-tradeoff) $\lambda$-tradeoff can be an accurate measure to select the optimal TA also for ManDisc; 2) (ManDisc v.s. ManDisc w/ once-for-all optimization) once-for-all optimization can achieve competitive (even slightly better) performance despite the parameter sharing among TA candidates; 3) (ManDisc w/ once-for-all optimization v.s. AutoDisc) the two together leads to results slightly better than those of ManDisc in a much more efficient manner.

    |Method|MRPC|QQP|Train Time|
    |--|--|--|--|
    |BERT_10%|87.8|84.6|1x|
    |ManDisc_10%|88.2|84.8|~20x|
    | w/ $\lambda$-tradeoff|88.2|84.8|~10x|
    | w/ once-for-all optimization|88.4|84.9|~10x|
    |AutoDisc_10%|88.4|84.9|~4x|

3. **Definition of *scale*.**

    We intend to mention it as the number of layers in Figure 1, and as the student/teacher parameter scale ratio in Figure 3. We will update Figure 1 in the revised paper so as that the *scale* consistently means the student/teacher parameter scale ratio.

4. **Sandwich-optimization is unclear.**

    We update the algorithm in the revised paper. To make it more clear, TAs are generated by pruning the teacher instead of a large TA.

5. **Optimal TA evolvement during the student distillation.**

    That's a good point. We plan to incorporate similar ideas so that the student can dynamically learn from all TA candidates, maybe similar to previous studies on multi-teacher distillation. However, currently multi-teacher would introduce additional memory consumption which is not in our expectation.

From: ICLR.cc/2023/Conference/Paper1589/Authors
title: Response to reviewer jzCn
comment: 1. **The scale of the teacher, teacher assistant, and student.**

    We make it clearer that the *scale* stands for the number of layers in Figure 1 and the student/teacher parameter scale ratio in Figure 3. We update Figure 1 in the revised paper.

2. **Sandwich-optimization is unclear.**

    To make the algorithm for once-for-all optimization clearer, we will update the algorithmic flow of once-for-all optimization in the revised paper. We currently present it with the following pseudo-code in a pytorch-like style. Note that here we use sparsity which equals to 1 - student/teacher parameter scale ratio in the paper. We certainly plan to release our code in the near future.

    ```python
        # given teacher t_model, train dataset train_data, evaluation dataset eval_data, initialize ta_model with copy().
        ta_model = copy(t_model)

        # specification.
        sparsities = [for x in range(0.05, 1.1, 0.05)]
        # subsample sparsities with eta.
        sparsities = subsample(sparsities, eta)
        sparsity_map = {}
        for s in sparsities:
            # obtain and store which parameters should be pruned for a given sparsity, with prune().
            sparsity_map[s] = prune(t_model, s)

        # optimization.
        for batch in train_data:
            t_output = t_model(batch)
            for s in sparsities:
                # obtain a candidate in-place with sparsify().
                sparsify(ta_model, s)
                ta_output = ta_model(batch)
                # compute loss and backward the loss.
                loss = loss_fn(t_output, ta_output)
                loss.backward()
            # update.
            optimizer.step()
            optimizer.zero_grad()

        # selection.
        # evaluate and store normalized performance of each with evaluate().
        performance = {}
        for s in sparsities:
            sparsify(ta_model, s)
            performance[s] = evaluate(ta_model, eval_data)
        tradeoff = {}
        for s in sparsities:
            tradeoff[s] = performance[s] + lam * s
        best_sparsity = sorted(tradeoff.items(), key=lambda x:x[1], reverse=True)[0]
        best_ta_model = sparsify(ta_model, best_sparsity)
    ```
 
3. **Differences between AutoDisc and Subformer.**
    
    The first difference is the styles of parameter sharing: AutoDisc shares parameters based on the incremental property of pruning while Subformer shares layers based on the manual setting. The second difference is the purposes: AutoDisc shares parameters to make it possible for once-for-all optimization while Subformer shares parameters only for reduction of parameters. Last but not least, the sandwiching in Subformer is very different from that in AutoDisc. We will add the discussion in the revised paper.

4. **Typos.**

    Thanks for pointing them out. We will fix them in the revised paper.

From: ICLR.cc/2023/Conference/Paper1589/Authors
title: General Response
comment: We would like to thank all the reviewers for their constructive comments. 
Specifically, we appreciate that 1) reviewers jzCn and PgbV agree that selecting the optimal TA is an important problem to study, e.g., reviewer PgbV points out that `This paper studies a very interesting problem, namely how to select useful teacher assistants without imposing significant computational requirements.`; 2) reviewer LkAS agrees that $\lambda$-tradeoff is good indicator in that `The paper proposes an indicator, i.e., -tradeoff to estimate the performance of a TA in knowledge distillation. Given a teacher and student, we can use the indicator to select the optimal TA efficiently.`; 3) all three reviewers agree that AutoDisc are efficient and effective, e.g., reviewer LkAS points out that `AutoDisc can achieve competitive performance compared with SOTA baselines, which significantly improved efficiency (~5-6x)`.

The main concern from the reviewers is that the clarity of the method. We attempt to make this paper clearer with some statements and supplementary experiments.

From: ICLR.cc/2023/Conference/Paper1589/Reviewer_jzCn
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a strategy for compressing large language models by identifying the “best” teacher assistant, which can help the “teacher” to teach the “student” model in the knowledge distillation process.  Comparing to the previous work of manually selecting the “best” teacher assistant, the proposed AutoDisc uses a more memory-efficient way to evaluate the candidates and conduct the selection, based on a defined evaluation criterion \lambda-Traddoff. 
strength_and_weaknesses: The strengths  of the paper:
- The study of compressing large language models is an important research problem, since powerful language models are getting bigger and bigger. 
- The AutoDisc can help preserve the capacity of large language models on small scale student models. 


The weaknesses of the paper:
- The paper is not easy to follow for readers out of the field of model compression. The key definition of some concepts are missing. For example, what is the “scale” of a teacher, teacher assistant and student model? 
- The parameter-sharing and sandwich-optimization are the key for reducing the computation cost in the step of “optimization”. However, there is no elaboration about how these ideas were implemented.  It is unclear how to conduct sandwich-optimization over the sampled candidates. 
- The ideas of parameter-sharing and sandwich-optimization have been used in language models [1]. How the ideas are different here in this paper?
- The paper has many writing errors, such as  “an one-layer”, “an one-run”,  “MSE, CE and KL stands for ”, “train time”

[1] Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers. EMNLP 2021, pages 4081–4090

clarity,_quality,_novelty_and_reproducibility: The paper needs to be improved by providing important information about the key concepts and the detail of methodology. For example, the scale of models, the implementation details of parameter-sharing and sandwich-optimization.  In addition, how the teacher assistant help the knowledge distillation process. 
summary_of_the_review: The value of this work is the presentation of an auto-selection strategy of teacher assistant for compressing large language models. The solution is based on parameter-sharing and sandwich-optimization for minimizing the evaluation process of candidate models. Although the idea seems interesting, the paper can be improved by addressing the weaknesses mentioned above.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1589/Reviewer_LkAS
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes the AUTODISC to automatically identify an effective teacher assistant (TA) for teacher assistant-based distillation. The authors introduce a $\lambda$-Tradeoff indicator to quantify the scale-performance tradeoff of the TAs and select the optimal one. Besides, the authors leverage once-for-all optimization to further reduce the training cost. Extensive experiments show that the proposed distillation schedule achieves comparable performance compared with the manual distillation schedule, but significantly improves the efficiency of ∼5-6×.
strength_and_weaknesses: *Strengths*

1. The paper proposes an indicator, i.e., $\lambda$-tradeoff to estimate the performance of a TA in knowledge distillation. Given a teacher and student, we can use the indicator to select the optimal TA efficiently. 
2. Experiments conducted on various NLU tasks and backbones demonstrate that AUTODISC can achieve competitive performance compared with SOTA baselines, which significantly improved efficiency (~5-6x).

*Weaknesses*

1. The basic assumption of the metric $\lambda$-tradeoff is not convincing, and it lacks theoretical and experimental support. 
    
    According to Sec2.1, the indicator $\lambda$-tradeoff for TA selection is based on an assumption, i.e., a TA with small scale and high performance brings a better student after distillation, but why? Given two TAs with comparable performance but different scales, does the student supervised by the smaller TA definitely outperform the student supervised by the other one? 

    The authors propose the assumption with little clarification. More theoretical or experimental support should be provided. Besides, the paper argues the $\lambda$-tradeoff and the student performance are positively correlated, but does not calculate the correlation coefficient based on extensive experiments, which also limits the reliability of the indicator.
    
2. Lack of two baselines: AUTODISC without sandwich-optimization and AUTODISC without $\lambda$-tradeoff selection.

    After specification, the framework AUTODISC contains two stages: 1. train a sandwich-like TA based on sandwich-optimization then separate it to multiple TAs, and 2. select the optimal TA based on $\lambda$-tradeoff. These two stages can degenerate into: 1. train multiple TAs separately, and 2. select the optimal TA based on the student performance after actual distillation. 

    In other words, it derives two baselines: AUTODISC without sandwich-optimization and AUTODISC without $\lambda$-tradeoff selection, which should be considered as an ablation study to verify the effect of the proposed optimization and selection strategies.
clarity,_quality,_novelty_and_reproducibility: Clarity: The writing should be improved:

1. Lack of formalized definition of the “scale“
    
    It takes considerable effort to understand Figure 1 and the methodology in Sec2 without a concrete definition of the “scale”. What does the term “scale" refer to? The number of model layers, the number of model parameters, or just a relative ratio? What does a teacher assistant with a scale of 8 mean in Figure 1? Why is the scale in Figure 1 a number larger than 1, but in Table 1, 2, and Figure 3 it is a number smaller than 1?
    
2. The introduction of samdwich-optimization is too vague

    If I have no misunderstanding, in practice, the AUTODISC train a large candidate TA, and then prunes its parameters to obtain multiple TAs with different scales. It would be better to clarify the procedure of the once-for-all optimization. 

Novelty:

1. The optimal TA may be changed as the evolvement of the student:
    
    The paper addresses that the optimal TA should be elaborately and automatically selected for a given teacher and student. Recent work [1] points out that the optimal teacher may be changed as the student competency increases during the distillation. I wonder if the optimal TA will also change during its distillation to the student.
    
    [1] Dynamic Knowledge Distillation for Pre-trained Language Models. Li et al, EMNLP 21
summary_of_the_review: The paper introduces the AUTODISC to significantly reduce the training cost in Teacher Assistant-based distillation. However, the proposed automatic indicator and assumption lack theoretical and experimental support. Besides, two baselines are also missed in the experiments. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1589/Reviewer_PgbV
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposes Automatic Distillation Schedule, which finds a good teacher assistant for model compression in a single trial. The authors discover that performance of the student model depends on the so-called $\lambda$-tradeoff, and further develop an algorithm that leverages such a metric. The proposed algorithm is divided into tree stages: specification, optimization and selection. Experiments on task-specific and task-agnostic distillation are provided to demonstrate the effectiveness of the proposed algorithm.
strength_and_weaknesses: Strengths:

* This paper studies a very interesting problem, namely how to select useful teacher assistants without imposing significant computational requirements. 

* The main results in the experiment demonstrate the effectiveness of the proposed algorithm.


Weaknesses and questions:

* Presentation of the paper:

  * This paper repeatedly uses the concept of “scale” without properly defining it. This makes it hard to understand Figure 1. Does a scale-5 model contain 5 times the number of parameters in a scale-1 model, or does the 12-layer BERT-base is defined as scale 10? Also, in Figure 3a, the scale becomes 0.1\~0.6 instead of 1\~6, please try to be consistent.

  * The “specification” process is easy to understand, but the presentation of “optimization” needs to be improved. For the optimization stage, suppose the candidate set is {5%, 10%, ... , 95%}, then how do we get the performance of all the candidates in a single run? The authors should probably provide a detailed algorithm in the appendix.

  * In the paragraph below Eq.1, what is the “feasible region” and where does “linear programming of the feasible region during selection“ come in? I do not see any linear programming part in the “selection” process. 

* The paper relies on the idea that the scale-performance tradeoff is a valid metric for student performance. The authors demonstrate this in Figure 1. However, I have several suggestions about the setup of Figure 1.

  * First, Figure 1 uses a one-layer student, which is not a practical setting. I suggest replacing the student with, for example, a 6-layer BERT model.

  * Second, Figure 1 only contains results on MRPC and RTE. These two datasets are very small and model performance usually has large variance. I suggest running experiments on larger datasets such as MNLI/QQP.

* The defined tradeoff has a hyper-parameter $\lambda$. This seems to introduce additional tuning efforts. I fail to see that performance of the proposed algorithm is stable with different values of $\lambda$. From Table 5, evidently performance drops by $0.6$ when $\lambda$ is slightly increased from $0.2$ to $0.3$. I suggest the authors to add more experiments, e.g., adding $\lambda=0.5, 0.8, 1.0$.

* I have concerns regarding the “optimization” stage of the algorithm. This stage seems to depend on the sandwich rule, which states that pruning more parameters leads to worse performance. However, I think the sandwich rule highly depends on the task (cv vs. nlp). In particular, the two cited papers (Yu & Huang, 2019; Cai et al., 2020) are for vision tasks instead of NLP. Also, in (Cai et al., 2020), it is argued that the sandwich rule holds when we decrease the width of the model, instead of pruning more parameters. In fact, in NLP, it is shown that pruning more parameters of BERT may even lead to better performance [1] (we can already see this in Figure 1 right).

[1] https://arxiv.org/abs/2105.12002

clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough