From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.
The response helped resolve some important concerns (e.g., strengthening of the experiments and their reproducibility).
The reviewers and meta reviewer all acknowledge that the submission proposes a principled construction of an ensemble, with appealing properties (e.g., anytime inference), within the context of distillation.

At this stage, while the submission does not suffer anymore from major flaws, there are still some arguably important aspects that warrant further consolidations/investigations, for instance:
* Improve the clarity of the exposition of the method (especially in Section 3.1.2)
* Stress-test the scalability of the approach (e.g., the matrices K’s grow with both the number of samples and classes — what happens for standard datasets like ImageNet-21k with ~12M points and 21k classes?)
* Further demonstrate the practical significance of the approach thanks to an actual, realizable gain in efficiency as directly measured by inference time (without using a conversion from FLOPs). Having a concrete implementation, compatible with modern hardware constraints (e.g., TPU with fixed-size batch sizes), is challenging for methods based on sequential-prediction and/or early-exit logic.   

Because of the extremely competitive landscape of the submissions this year, the paper remains under the cut and has ultimately not been selected for acceptance.
We are convinced that the suggestions above will help strengthen the paper for a future resubmission, which the reviewers and meta reviewer all encourage.

justification_for_why_not_higher_score:  * The practical significance of the approach is not sufficiently vetted and demonstrated:
   * Scalability
   * More concrete evidence of the resulting gain in inference time.
* Exposition of the method lacks clarity.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5246/Authors
title: Response to Reviewer 3 (GAfE)
comment: We wanted to check if your concerns have been satisfactory answered; If there are more queries please let us know so that we can address them. Thank you for your time.

From: ICLR.cc/2023/Conference/Paper5246/Authors
title: Response to Reviewer 3 (GAfE)
comment: We thank the reviewer for their detailed review and comments. Here we address each of the concerns raised in the review:

1. *Baselines and experiments*

> For a given efficiency ensemble latency, find a model architecture that has similar latency and compare accuracy. It’s possible you do this in Figure 5, but I did not find that figure particularly easy to understand.

We have improved our presentation in the experimental section. The baseline NO-RESHED in Figure 2 now corresponds to the baseline the reviewer recommends.  We have removed the previous Figure~5 in in the new presentation.

2. *Larger scale datasets*

> CIFAR10 and SVHN are not particularly convincing benchmarks. I would encourage the authors to attempt to scale their experiments to at least CIFAR100 or Tiny-ImageNet.

We have included the results with CIFAR100 image classification in place of SVHN dataset. 

3. *Student model details*

> You mention that student models have “significantly fewer parameters and resource requirements”. This is very vague and needs to be quantified.

We have added the exact base model details, along with hyperparameters we used as part of appendix D. 


We hope the above changes have addressed the reviewers' concerns and questions. Please let us know if you have any additional questions.

From: ICLR.cc/2023/Conference/Paper5246/Authors
title: Response to Reviewer 2 (9Tcs)
comment: We thank the reviewer for their detailed review and comments. Here we address each of the concerns raised in the review:

1. *Baselines* 

> The description of the experimental baselines should be much clearer.

We have improved our explanation of the baselines in the experimental section. We also now present separate experiments for early-classification and any-time inference settings to contextualize B-DISTILL better. In the early-classification experiments, we compare to the E-RNN method from (Dennis et al., 2018). 

2. *Clarity and figures*

> The Figures (and Figure 3 in particular) are difficult to view (due to size, even on a screen and zooming in), and to understand in general. Am I correct in understanding that round reports the FLOP percentage for an ensemble of size (assuming each round of FIND-WL succeeds)? For the FLOP analysis, can you confirm that distinction between fixed-model and B-Distill is inconsequential?

 We have revised all our plots to make them easier to view. In the current version, Figure 2, reports inference time on the x-axis. Figure 3 reports the FLOPs required for the model added in round $T$ (not the ensemble). Summing over all $T$ in Figure 3 provides the FLOPs required for the ensemble.

3. *Reproducibility*
> The paper appears mostly reproducible; however, the authors should include the detailed architectures used as base learners in all experiments.

We have included the base classes used for all the experiments and training recipes with all hyperparameter configuration to Appendix D.


4. *Other questions*

> Could you clarify why you need to split the matrices K into K+ and K−?

We require this split because we bound $|f-g|$ by bounding $f-g$ and $-(f-g)$ separately. In the AdaBoost setup, since the labels are always in $\{-1, 1\}$, for a classifier $h$ and label function y, $yh = 1$ whenever $y(x) = h(x)$ and $-1$ when $y(x) \ne h(x)$. Thus the classification error can be written as $0.5 * (1 - yh)$ which takes the value 0 when $h$ correctly classifies $y$ and $1$ otherwise. Since this value is always in $\{0, 1\}$, a one sided bound suffices to bound the loss away from $1$. This corresponds to the one-set of weights in the AdaBoost case. In contrast, in our case since $f \in [-1, 1]$ after normalization (for each label), $-2 \le f-g \le 2$, and we need a two sided bound to argue that $|{f - g}| \le \epsilon$, one side arguing that $ f - g \le \epsilon $ and other side for $(f-g) \ge -\epsilon$.

> I'm confused about why p in (3) has is of shape N×L. I understand that N is for the distribution over samples of the dataset; is the L dimension to reweight the loss for different logit dimensions on each sample?

Yes you are correct. Although we don’t use a classifier per label, our loss function still penalizes each label separately. This is so that we are consistent with our theoretical setup. The convergence bound in theorem 1 is in terms of individual labels $j$ and reflects this choice of shape.


> In Theorem 2, is μ a distribution (and the inequality stochastic dominance), or a scalar?

$\mu$ is a scalar here.

> Experimentally, how often did FIND-WL fail?

There are two failure modes for FIND-WL. The first is when FIND-WL does not find any weak learners in the expanded function class. This rarely happens and FIND-WL almost always finds weak-learners from the expanded classes. The second failure mode happens when FIND-WL finds a weak learner, but it improves the ensemble performance only by small amounts. Towards the later boosting rounds, this failure mode can be an issue. A simple fix is to expand with slightly more expensive connection towards later rounds.  (See Appendix D for base model architecture details).

> Is there an extraneous index in (3)

Yes. Thank you for catching it; it has been fixed.

We hope the above changes have addressed the reviewers' concerns and questions. Please let us know if you have any additional questions.

From: ICLR.cc/2023/Conference/Paper5246/Authors
title: Response to Reviewer 1 (kWeE)
comment: We thank the reviewer for their detailed review and comments. Here we address each of the concerns raised in the review:

1. *Clarity of section 3:*

 > I think section 3 in particular was structured in a way that was hard to parse at first.

 We have modified the section and have attempted to improve the presentation of the section. 

2. *Reproducibility:*

> The paper and appendix do not provide enough information to reproduce the experiments, such as chosen hyperparameters or exactly how the smaller capacity models were constructed.

 Base classes used for all the experiments and training recipes with all hyperparameter configuration have been added to Appendix section D. We will also release our experimental code once anonymity restrictions have been lifted.

3.*Basic boosting algorithm:* 

 > I think the evaluation would be more convincing if it considered more baselines such as a basic boosting algorithm.

 We have added baseline comparison with AdaBoost adapted for distillation in Figure 2. Here, instead of training against the labels as in regular AdaBoost, we train against the target function (teacher model). The appropriate update rule for the distribution over weight can be computed through the same argument as in Section 3, Equation~2 about weak gradients. 

We hope the above changes have addressed the reviewers' concerns and questions. Please let us know if you have any additional questions.

From: ICLR.cc/2023/Conference/Paper5246/Authors
title: Overview of Revisions
comment: 
We thank the reviewers for their detailed review and suggestions for improving the paper. Since all three reviewers recommended changes and improvements to the experimental section, we present an overview of the changes made in the revision here. We hope that these have addressed the reviewer’s concerns. Please let us know if you have any additional questions or if all the issues have been addressed.


1. Improved presentation in the experimental section and added better explanations for baselines. Some results previously in the experimental section moved to the appendix for clarity.
2. Experiments are now presented in the context of anytime inference and early-exit. 
3. Added baseline comparisons to the E-RNN algorithm, the early-exit method from (Dennis et al., 2018).
4. Added baseline comparisons to AdaBoost (modified for distillation).
5. Added experimental results for the relatively large scale CIFAR100 image classification dataset (replacing SVHN).
6. Added experimental details to the appendix for reproducibility: base-model class information, connections used and training hyperparameters included. 
7. Improved presentation in section 3 through minor edits in the text.

From: ICLR.cc/2023/Conference/Paper5246/Reviewer_kWeE
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the problem of progressive distillation, which is defined as approximating a large model with an ensemble of multiple small models. The paper poses this problem as a two-player zero-sum game where a “distribution player” poses a distribution over example assignments to learners to maximize loss, and the “hypothesis player” chooses hypotheses for each learner to minimize loss.
strength_and_weaknesses: The problem setting is interesting and well-motivated. Standard distillation (one big model to one small model) is likely to result in underfitting, whereas considering an ensemble of small models increases the capacity while ideal latency can be kept low through parallelization.

The intermediate layer connection idea (section 3.1.2) is interesting - instead of just considering a bigger model class, you can start from embeddings from previous generation nets; this is probably better for when you want to do early-exit or anytime inference with limited compute.

In experiments, the paper reports benefits in six different datasets, each with different network architectures. However, I think the evaluation would be more convincing if it considered more baselines such as a basic boosting algorithm.

Minor comment: the paper says “SVNH dataset” repeatedly; is this a typo for SVHN (Street View House Numbers)?
clarity,_quality,_novelty_and_reproducibility: The writing wasn't perfect, but clear enough to understand the main points. I think section 3 in particular was structured in a way that was hard to parse at first. To my knowledge, the problem setting and approach is novel. The paper and appendix do not provide enough information to reproduce the experiments, such as chosen hyperparameters or exactly how the smaller capacity models were constructed.
summary_of_the_review: I think the problem setting and method is interesting, but the experimental evaluation is somewhat weak.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5246/Reviewer_9Tcs
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper introduces a method (B-DISTILL) to distill an (expensive) trained model into an ensemble of weak learners. These learners are meant to provide efficient inference, with progressively better results as the ensemble size increases: this allows for a straightforward way to trade off accuracy and compute at inference time, in cases where inference may be interrupted abruptly, or when variable levels of accuracy are expected.

The distilled ensemble is obtained by solving a two player zero-sum game, with convergence and generalization bounds provided under weak assumptions. In order to the $n+1$-th model to refine the prediction of the previous $n$ models, the authors also allow each successive model to _reuse_ the feature transformation of the previous ensemble member.

Experimentally, authors verify their method on simulated, vision, speech, and sensor-data classification tasks, and across a variety of architectures.
strength_and_weaknesses: # Strengths
- The paper is very well written, and the exposition is overall clear (aside from issues with baselines and $K$ matrices mentioned below)
- The method has a clear application and is of similarly clear value to the community (anytime and efficient inference).
-  The authors provide convergence guarantees ($\mathcal O(\sqrt{\ln 2N / T})$ for $T$ the ensemble size and $N$ the dataset size)

# Weaknesses
- The description of the experimental baselines should be much clearer. As it stands, I found it difficult to be certain whether the authors consider Fixed-model to be an ensemble (which appears to be the case), and how successive ensemble members differ from each other.
- It does not appear that the authors included early exit baselines; it would strengthen the paper significantly to compare to work such as (Dennis et al., 2018), despite the fact that such work is not distillation based.
- The Figures (and Figure 3 in particular) are difficult to view (due to size, even on a screen and zooming in), and to understand in general. Am I correct in understanding that round $T$ reports the FLOP percentage for an ensemble of size $T$ (assuming each round of FIND-WL succeeds)? For the FLOP analysis, can you confirm that distinction between fixed-model and B-Distill is inconsequential?

# Questions to the authors
- Could you clarify why you need to split the matrices $K$ into $K^+$ and $K^-$?
- I'm confused about why $p$ in (3) has is of shape $N \times L$. I understand that $N$ is for the distribution over samples of the dataset; is the $L$ dimension to reweight the loss for different logit dimensions on each sample?
- In Theorem 2, is $\mu$ a distribution (and the inequality stochastic dominance), or a scalar?
- Experimentally, how often did FIND-WL fail?

# Nitpicks
- Please add a $y$-axis to Figure 5. Are you reporting accuracy?
- Consider moving the legend above the sub-figures when possible, to improve readability. 
- Is there an extraneous index $j$ in (3)?
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Overall, the paper is clear, although the current discussion surrounding the probability matrices $K$ and the exact setup for the baselines is insufficient. Should these issues be addressed, there would be no issues with clarity.

**Quality**: The results seems of good quality (assuming my understanding of the baselines is correct).

**Novelty**: The work is fairly novel, although its experimental results should be placed more clearly within the context of early exiting/anytime inference.

**Reproducibility**: The paper appears mostly reproducible; however, the authors should include the detailed architectures used as base learners in all experiments.


summary_of_the_review: This paper addresses an interesting problem (distilling a model into an ensemble of weak learners to control inference cost and accuracy trade-offs). Taking a game theory approach, authors introduce an algorithm with convergence guarantees, and show experimentally that their approach successfully trades off computational costs with model accuracy. 

However, I found it quite difficult to understand the detail of the experimental results, and of the baselines in particular. This work would be very much strengthened by a clearer experimental section, and the inclusion of baselines from the multiple-instance learning / anytime inference literature.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5246/Reviewer_GAfE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose the problem of progressive distillation: approximating a large model with an ensemble of smaller models. They seek a solution that \emph{decomposes} the large model, such that the quality of the approximation improves monotonically with the size of the ensemble. They formulate progressive distillation as a two player zero-sum game and present B-DISTIL, an algorithm for solving this game. One notable feature of their algorithm is that it allows models to reuse intermediate representations from other models in the ensemble. They evaluate their algorithm on image classification (ResNets on CIFAR10 and DenseNets on SVHN), speech classification (LSTM on Google-13), and sensor data (GRU on DSA-19). They find that their B-DISTIL-generated ensembles allow favorable tradeoffs between computation time and accuracy.
strength_and_weaknesses: **Strengths**

Good motivation

Clever method—reusing intermediate representations within an ensemble is an elegant idea.

**Weaknesses**

Experiments are not very convincing; baselines and benchmarks are insufficient.

Minor clarity issues.

clarity,_quality,_novelty_and_reproducibility: **Empirical Rigor**

In general, the experiments feel like a bit of an afterthought.

The experiments are lacking baselines. What are the current best methods to reduce latency at the cost of reduced accuracy? One simple baseline that comes to mind is training a smaller model from scratch. For a given efficiency ensemble latency, find a model architecture that has similar latency and compare accuracy. It’s possible you do this in Figure 5, but I did not find that figure particularly easy to understand.

CIFAR10 and SVHN are not particularly convincing benchmarks. I would encourage the authors to attempt to scale their experiments to at least CIFAR100 or Tiny-ImageNet.

**Clarity**

You mention that student models have “significantly fewer parameters and resource requirements”. This is very vague and needs to be quantified.

X- and Y-axes in Figure 5 need labels.

summary_of_the_review: Clever idea, needs more rigorous empirical evaluation. Could be publishable with more thorough experimentation.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough