Paper: InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning (ICLR-2024)

Reviewer comment (Novelty):
**originality**: The idea is somehow similar to other dynamic pruning approaches. It does not provide a different point of view in the matter.

Author response:
**A1:** Thanks for the comment. In section 1 paragraph 5, we stated: 
* "Directly pruning data may lead to a biased gradient
estimation as illustrated in Fig. 1a, which affects the convergence result. This is a crucial factor that
limits their performance, especially under a high pruning ratio (corresponding results in Tab. 1)"

To better illustrate the difference, we show a table here for comprehensive analysis:

| method                       | soft prune | rescale | anneal | unbiased| Task                                         | lossless saving ratio |
|------------------------------|------------|---------|--------|---------|----------------------------------------------|------------------------|
| Other Dynamic Pruning [1][2] | no         | no      | no     | no      | Classification                               | ~20%                   |
| InfoBatch                    | yes        | yes     | yes    | yes     | Classification, Segmentation, Diffusion, LLM | 40%                    |


**Analysis:** 
* Previous methods don't use soft pruning. It could lead to a biased gradient direction (see Appendix B.1, or B.2 in the updated revision).
* Previous methods don't use rescaling. This leads to a lack in total update, which is more severe at a higher pruning ratio.
* Previous methods don't use annealing. Therefore a remaining bias could be left (Appendix B.2 or updated revision B.3 further discussed annealing).
* Previous methods mainly focus on classification. InfoBatch can be applied to a broader range of tasks.


**Conclusion:** 
InfoBatch proposes a **probabilistic framework** aiming to achieve unbiased dynamic pruning (with soft pruning, rescaling, and annealing), which differs from
other dynamic pruning works that propose **metrics** and use **sort** to select more important samples.

Reviewer comment (Experiments):
**quality**: In the ablation study, I would like to see if different threshold selections (apart from the mean value) can affect the algorithm. I find it a little bit odd that there is no discussion regarding to this point.

Author response:
**A2:** Thanks for the comment. We conduct these ablations on CIFAR-100 ResNet-50 as follows:

**Setting:**  All results are averaged across three runs with std reported.
We report their acc and pruning ratio here.

| threshold      | Acc. (%) | Prune. (%) |
|----------------|----------|------------|
| <mean(default) | 80.6±0.1 | 32.8       |
| <mean+0.5*std  | 80.2±0.1 | 35.4       |
| <mean+ 1*std   | 80.1±0.1 | 38.2       |
| <mean+ 2*std   | 80.1±0.1 | 41.5       |
| <75%           | 80.6±0.1 | 32.8       |
| <85%           | 80.3±0.1 | 36.9       |

**Analysis:** 
* mean + c*std as a threshold with c>0 can achieve a higher saving ratio but with degraded performance.
* percentile threshold selection can also lead to a reasonable performance. 
* Specifically, the <75% threshold gets a similar saving ratio and performance as using mean. 
* <85% threshold achieves both higher saving ratio and performance than 
mean+0.5*std

**Discussions:**
* According to the inference in 2.3, the unbiasedness doesn't depend on the threshold selection. **The experiment result matches the theoretical analysis**
as using the 75 percentile threshold also gives a lossless result.
* std factor is not preferred, properly because entropy loss is not Gaussian and long-tailed at convergence 
(we add a visualization in revised Appendix E.1).
* if assuming Gaussian distribution for some other metrics behind loss, percentile of loss can do the same thing as using means + c*std.

**Conclusion:** InfoBatch is robust to different threshold selection methods. Empirically, the std factor is not preferred for loss. Percentile thresholds could also be a choice.

Reviewer comment (Presentation):
**clarity**: The text is too dense. The figures are too small to be read in a paper. I suggest the authors to increase the figures, while removing the text surrounding them. Some sections, like 2.3, can be summarized to make room for the adjustment.

Author response:
**A4:** Thanks for the comments. We shortened the 2.3 and rearranged the layout to enlarge the figures (in the updated revision). The corresponding change 
is as follows:
* Sec 1, we enlarge Fig 1 and move it to the top of the page.
* Sec 2.3, we shorten the Theoretical analysis and move part of the proof to B.1.
* Sec 3, we put the previous Fig 3 and Fig 4 together to the top, making them no longer surrounded by text.
* Sec 3.5, we increase the size of Table 8 and Table 9.

Reviewer comment (Novelty):
**significance**: the authors claim their threshold value can be established in constant time, whereas the state-of-the-art methods require a sorting part (the complexity should be $\mathcal{O}(N \log N)$. However, I think this improvement is no significant, as the speedup produced is residual compared to the time needed to train the network.

Author response:
**A5:** Thanks for the question. Sort is O(NlogN) for full data and amortized O(logN) for each sample, ours is O(N) and 
amortized O(1). We design the operation considering it could differ in a large factor when N is very big, as shown in Appendix D table 11.

The overhead saving of operation compared to UCB is 10 times (0.03h to 0.0028h) on ImageNet-1K; however we agree it is 
not a significant improvement compared to training time saving (17.5h->10h). Our main saving comes from saving the forward and backward
computation, which is shown in the table below:

**Setting:** ResNet-50 on CIFAR-100. All results are averaged across three runs with std reported.
We report their acc and pruning ratio here. 

|           | Overhead (s) | Saving      | Acc. (%) |
|-----------|--------------|-------------|----------|
| UCB       | < 2          | 20%, ~800s  | 80.4±0.2 |
| UCB       | < 2          | 33%, ~1320s | 79.8±0.2 |
| UCB       | < 2          | 41%, ~1640s | 79.5±0.2 |
| InfoBatch | < 0.1        | 33%, ~1320s | 80.6±0.1 |
| InfoBatch | < 0.1        | 41%, ~1640s | 80.6±0.2 |

**Analysis**: 
* At lossless performance of ResNet-50 on CIFAR-100, InfoBatch can save 41% cost compared to 20% of UCB. The saved time is a 100% (800s->1600s) improvement.
* At the same saving ratio, UCB has a much higher performance degradation

We degraded the claim of the part on time complexity, which can be visible in the new revision. The main modifications can 
be summarized as:
1. in section 4 related works, we change "which could be a non-negligible overhead" -> "which could be an overhead"

Reviewer comment (Experiments):
Why there is no discussion regarding to the threshold selection procedure? Different solutions like taking the mean plus a factor of the standard deviation can led to interesting results.

Author response:
**A2:** Thanks for the comment. We conduct these ablations on CIFAR-100 ResNet-50 as follows:

**Setting:**  All results are averaged across three runs with std reported.
We report their acc and pruning ratio here.

| threshold      | Acc. (%) | Prune. (%) |
|----------------|----------|------------|
| <mean(default) | 80.6±0.1 | 32.8       |
| <mean+0.5*std  | 80.2±0.1 | 35.4       |
| <mean+ 1*std   | 80.1±0.1 | 38.2       |
| <mean+ 2*std   | 80.1±0.1 | 41.5       |
| <75%           | 80.6±0.1 | 32.8       |
| <85%           | 80.3±0.1 | 36.9       |

**Analysis:** 
* mean + c*std as a threshold with c>0 can achieve a higher saving ratio but with degraded performance.
* percentile threshold selection can also lead to a reasonable performance. 
* Specifically, the <75% threshold gets a similar saving ratio and performance as using mean. 
* <85% threshold achieves both higher saving ratio and performance than 
mean+0.5*std

**Discussions:**
* According to the inference in 2.3, the unbiasedness doesn't depend on the threshold selection. **The experiment result matches the theoretical analysis**
as using the 75 percentile threshold also gives a lossless result.
* std factor is not preferred, properly because entropy loss is not Gaussian and long-tailed at convergence 
(we add a visualization in revised Appendix E.1).
* if assuming Gaussian distribution for some other metrics behind loss, percentile of loss can do the same thing as using means + c*std.

**Conclusion:** InfoBatch is robust to different threshold selection methods. Empirically, the std factor is not preferred for loss. Percentile thresholds could also be a choice.

Reviewer comment (Experiments):
Did the authors experiment with a pruning probability that depends on the sample score?

Author response:
**A3:** Thanks for the question. To improve data efficiency, we use the percentile method to prune part of the data (20%) with a more aggressive ratio r (0.75)
as the default setting on several datasets, as claimed in section 3.1.

By doing so, a higher pruning ratio can be achieved without performance loss. In Table 6 with this improvement,
CIFAR-100 ResNet-50 can achieve 41.3% saving ratio, compared to 33% in Table 5 with uniform r.

A theoretical related analysis is in Appendix B.3 (updated revision B.4), suggesting that $\mathbb{E}\_\text{rescaled}[G\_{z}^2/(1-\mathcal{P}\_t(z))]\leq {\mathbb{E}}\_{\mathcal{D}}[G_z^2]$
could be a condition for healthy rescaling. This indicates that we can prune and rescale more aggressively for 
samples with low loss (gradients).

To better illustrate the effect of using different pruning probabilities depending on the sample score,
we verified different combinations as follows:

**Setting:** InfoBatch on CIFAR-100 ResNet-50 with different thresholding method. 
All results are averaged across three runs with std reported.
We report their acc and pruning ratio here.

| range:ratio(r)                                                         | saving ratio (%) | performance (%)  |
|------------------------------------------------------------------------|--------|--------------|
| full data (r=0)                                                        | 0      | 80.6±0.1     |
| <mean:0.5                                                              | 33     | 80.6±0.1     |
| [0%,20%):0.75,[20%,85%):0.5                                            | 41.3   | **80.6±0.2** |
| [0%,30%):0.75,[30%,90%):0.5                                            | 45     | 80.4±0.2        |
| [0%,36%):0.75,[36%,95%):0.5                                            | 49     | 80.3±0.2        |
| [0%,20%):0.75，[20%,40%):0.7，[40%,60%):0.65，[60%,80%):0.6，[80%,90%):0.5 | 51.3   | 80.1±0.2         |


**Analysis:** 
* Adaptive r with [0%,20%):0.75,[20%,85%):0.5 gets the 41.3% saving ratio with lossless performance. 
* Adaptive r depending on the score could require a corresponding parameter search.
* Moving thresholds rightward would cause degraded performance.

**Conclusions:**
* A pruning probability depends on sample score could further improve data efficiency.
* It is not straightforward to find a better adaptive r with manually tuning

**Future work:** **To easily find adaptive r values,** we plan to propose a cheap estimator for gradient G and try setting adaptive r based on it ensuring $\mathbb{E}\_\text{rescaled}[G\_{z}^2/(1-\mathcal{P}\_t(z))]\leq {\mathbb{E}}\_{\mathcal{D}}[G_z^2]$.

Reviewer comment (Experiments):
The comparison with the baselines does not seem entirely apples-to-apples to me due to the "annealing" period on the whole dataset. I suspect (intuitively and based on the ablations in Table 4 plus the pruning rule seemingly being irrelevant in Table 5) that ensuring the total length of the optimization trajectory remains comparable to that on the full dataset (by rescaling the gradients) in conjunction with the fine-tuning on all data towards the end of training is the "secret sauce" to making a dynamic pruning method perform without a degradation in test accuracy. I'm not familiar with the (Raju et al., 2021) paper, but would expect that at least the annealing period could be incorporated into this method without further issue. At the moment the paper presents its selection rule leading to performance matching that of full-data training as a core contribution, however if this can be achieved relatively easily with other selection techniques as well, I think it becomes more about the re-scaling/tuning as general purpose techniques and the cost comparison between different selection approaches being featured more prominently. I don't think this would worsen the paper at all, although it would change the key takeaways a fair bit and I think it is important that the latter accurately reflect the empirical results.

Author response:
**Q1: Are rescaling and annealing all the key points of InfoBatch?**

**A1:** Thanks for the question. This is partially right. **Only with soft (probabilistic) pruning, recaling can take effect
with no bias.** Rescaling for hard pruning would harm the performance. Annealing can be used in all settings, which helps to 
achieve lossless performance when rescaling is unbiased. To verify this, we conduct the following experiment on CIFAR-100 ResNet-50.

| method                     | saving ratio  (%)| performance (%) |
|----------------------------|------------|-------------|
| InfoBatch w/o soft pruning | 33         | 80.1        |
| InfoBatch w/ soft pruning | 33         | 80.6        |

Note: training on the whole dataset achieves 80.6% acc.

**Conclusion: Soft pruning is important to unbiased rescaling and achieving lossless results.**

This question is not fully answered yet at this point. The further insight of InfoBatch's sample selection is discussed 
in answer A2 and A3.

Reviewer comment (Experiments):
On a related note, I am a little bit concerned that the hyperparameters on e.g. ResNets for CIFAR10 are not tuned for achieving the final test performance as quickly as possible. I think it would be worth adding a baseline that trains on all data with a reduced number of epochs/learning rate decay milestones but increased learning rate corresponding to the computation saved by pruning (so hypothetically for 20% saved computation, train for 80 instead of 100 epochs but with learning rate 1.25 instead of 1 and halve it after 40 instead of 80 epochs). This is to ensure that pruning approaches meaningfully speed up training rather than benefitting from slack in the canonical hyperparameter choices for benchmark problems.

Author response:
**Q4: Add a full data baseline with reduced epochs and adjusted learning rate.**
**A4:** Thanks for the question. Here we add the tuned learning rate baseline with reduced epoch numbers. 


| settings with adjusted learning rate         | saving ratio      | performance (%) | InfoBatch performance |
|----------------------------------------------|-------------------|-----------------|-----------------------|
| CIFAR-10 ResNet-18 baseline     | 30%   (140 epoch) | 94.8            | 95.6 ⭡ 0.8            |
| CIFAR-10 ResNet-18 baseline     | 50%   (100 epoch) | 94.6            | 95.1 ⭡ 0.5            |
| CIFAR-10 ResNet-18 baseline     | 70%   (60 epoch)  | 92.7            | 94.7 ⭡ 2.0            |
| CIFAR-100 ResNet-18 baseline    | 30%   (140 epoch) | 77.0            | 78.2 ⭡ 1.2            |
| CIFAR-100 ResNet-18 baseline    | 50%   (100 epoch) | 76.9            | 78.1 ⭡ 1.2            |
| CIFAR-100 ResNet-18 baseline    | 70%   (60 epoch)  | 76.3            | 76.5 ⭡ 0.2            |
| ImageNet-1K ResNet-50 baseline  | 60%   (54 epoch)  | 72.8            | 76.5 ⭡ 3.7            |

**Analysis** 
* InfoBatch outperforms the tuned baseline. This is properly because higher loss samples are 
more sensitive to learning rate scaling as discussed in Appendix B.3. 
* Based on that, when scaling the learning rate to match
the original training progress, scaling lower loss samples is still preferred than higher loss samples. 

[//]: # (We add this baseline to Table 1 in the revision.)

**Conclusions:**
* With the same computation, InfoBatch has higher performance than the tuned baseline.
* According to the table and the point above, the improvement of InfoBatch is not caused by a lack of tuning of the benchmark.

Reviewer comment (Theory):
I don't entirely follow what the theoretical analysis is trying to achieve in 2.3. Isn't this just showing that the soft-pruned and rescaled gradient is unbiased? Isn't this completely obvious from having independent Bernoullis multiplied onto the terms of a sum (the total gradient over the dataset) and the expectation of a Bernoulli being its probability (so that if we divide by the probability, we get an expectation of 1 and the sum remains unchanged)?

Author response:
**Q5: Purpose of 2.3**

**A5:** Thanks for the question. Previously, considering the different backgrounds of reviewers, we wrote 2.3 with more detailed 
steps. We summarize it to make it brief (in the updated revision), the changes are as follows:

* We merge previous Eqn. (6) and (7) to revised Eqn. (6), and move the middle step to B.1.
* We merge previous Eqn. (8) and (9) to revised Eqn. (7).

Reviewer comment (Writing):
I found the paper to be fairly different to what the title made me expect. "Info" and "lossless" imply a connection with information theory and lossless compression to me, which is of course not present in the method. I appreciate that this is entirely subjective, but would suggest reconsidering the title of the paper. In particular, I would argue that the "lossless" part is a bit misleading since this is not theoretically guaranteed by the method, but merely an empirical observation in the experiments. Of course matching performance to the full dataset can always be achieved by letting \( r \rightarrow 0, \delta \rightarrow 1 \), but this would remove any cost savings.

Author response:
**Q6: "Info" and "lossless" in the title**

**A6:** Thanks for the comment. 
* We call our method "InfoBatch" because it takes advantage of the current learning status (loss) to do selective data pruning. It is "informative"
instead of purely random.
* We achieve lossless results on many tasks with substantial savings, which is an important milestone. It is true InfoBatch cannot
guarantee a lossless result in all settings, but the empirical hyperparameters have already achieve lossless results on image classification,
semantic segmentation, image generation, and language model finetuning.

We are welcome to further discussion on this problem during the author reviewer discussion period.

Reviewer comment (Novelty):
Similarly, I think the paper overstates its relationship with coreset/data selection methods a little bit. These are generally not for speeding up an initial training run, but subsequent ones e.g. for continual learning or hyperparameter tuning and typically incur an upfront cost. On the contrary, the proposed method speeds up a given training run without producing any artefacts (a coreset) that are useful downstream. So to me this seems more like a curriculum learning paper (although I am not particularly familiar with this branch of the literature, so this is a somewhat speculative statement).

Author response:
**Q9: Relation to curriculum learning**
**A9:** Thanks for the comment. Our work shares some similarities with curriculum learning while differing in many aspects.
We summarize it as follows:

| feature\method                            | InfoBatch                               | Curriculum Learning ([1] as an example)                     |
|-------------------------------------------|-----------------------------------------|-------------------------------------------------------------|
| Calculation before training               | No                                      | Yes                                                         |
| Change sample frequency                   | Yes                                     | Yes                                                         |
| Arrange method                            | Statistical                             | Sort                                                        |
| Scoring Overhead                          | O(1) per sample, obtained from training | At least one forward pass per sample (O(M)) before training |
| Overall Cost                              | Negligible                              | Non-negligible                                              |
| Purpose                                   | Acceleration                            | Performance Improvement                                     |
| Adaptive to learning status               | Yes                                     | No                                                          |
| Pruned Part                               | Easy                                    | Hard (Standard)                                             |
| CIFAR-100 ResNet-50 Saving at Convergence | 40%                                     | 10%                                                         |

[1] Wu, Xiaoxia, Ethan Dyer, and Behnam Neyshabur. "When do curricula work?." arXiv preprint arXiv:2012.03107 (2020).

Reviewer comment (Experiments):
I would like to see results for Random*, \(\epsilon\)-greedy and UCB with annealing and gradient re-scaling (for Random*; for the other two as applicable). As much as possible for Table 1 and ideally Table 2 (ResNet-18 instead of 50 is perfectly fine if that makes it more realistic). My hypothesis here would be that all baselines will match InfoBatch in discriminative performance, which would necessitate the main claims in the paper being updated (again, I don't think this affects the value of the contribution). If all the baselines are already using annealing and rescaling, this point is of course void.

Author response:
**Q2: (Generalization) Results for other dynamic baselines with annealing and rescaling**
**A2:** Thanks for the question. The random in Table 5 (CIFAR-100 ResNet-50) is the random* with annealing and rescaling.
Its result is improved as expected over table 4, since random* is also a kind of soft pruning. 


However on UCB which uses (sort and) hard pruning, the thing is different. The rescaling can only be applied to all remaining 
samples which are all higher score samples, being statistically different from pruned samples. 
We show the corresponding experimental results here:

Setting: Train ResNet-50 on CIFAR-100 using UCB and random*. We report their acc and pruning ratio here.

| method                                      | saving ratio (%) | performance  (%) | compared to InfoBatch |
|---------------------------------------------|--------------|--------------|-----------------------|
| UCB                                         | 33%          | 79.9         | -0.7                  |
| UCB                                         | 41%          | 79.5         | -1.1                  |
| UCB+anneal                                  | 33%          | 80.1         | -0.5                  |
| UCB+anneal                                  | 41%          | 79.6         | -1.0                  |
| UCB+rescale+anneal                          | 33%          | 79.9         | -0.7                  |
| UCB+rescale+anneal                          | 41%          | 79.5         | -1.1                  |
| random*                                     | 33%          | 79.7         | -0.9                  |
| random*+rescale+anneal<br/>(Table 5 random) | 33%          | 80.5         | -0.1                  |
| random*+rescale+anneal                      | 41%          | 80.3         | -0.3                  |


**Analysis:**
* UCB's performance is increased by annealing using the same saving ratio (by controlling the pruning fraction), but much lower than
InfoBatch
* Rescaling with annealing improves the performance of random* using the same saving ratio; but adding rescaling leads to degraded performance
of UCB

**Conclusion:** **Annealing could help to improve performance on a broader range of
sample selection methods, but rescaling can only take effect if it is using soft pruning.**

Reviewer comment (Experiments):
Add a full data baseline with reduced epochs as proposed in the weaknesses.

Author response:
**Q4: Add a full data baseline with reduced epochs and adjusted learning rate.**
**A4:** Thanks for the question. Here we add the tuned learning rate baseline with reduced epoch numbers. 


| settings with adjusted learning rate         | saving ratio      | performance (%) | InfoBatch performance |
|----------------------------------------------|-------------------|-----------------|-----------------------|
| CIFAR-10 ResNet-18 baseline     | 30%   (140 epoch) | 94.8            | 95.6 ⭡ 0.8            |
| CIFAR-10 ResNet-18 baseline     | 50%   (100 epoch) | 94.6            | 95.1 ⭡ 0.5            |
| CIFAR-10 ResNet-18 baseline     | 70%   (60 epoch)  | 92.7            | 94.7 ⭡ 2.0            |
| CIFAR-100 ResNet-18 baseline    | 30%   (140 epoch) | 77.0            | 78.2 ⭡ 1.2            |
| CIFAR-100 ResNet-18 baseline    | 50%   (100 epoch) | 76.9            | 78.1 ⭡ 1.2            |
| CIFAR-100 ResNet-18 baseline    | 70%   (60 epoch)  | 76.3            | 76.5 ⭡ 0.2            |
| ImageNet-1K ResNet-50 baseline  | 60%   (54 epoch)  | 72.8            | 76.5 ⭡ 3.7            |

**Analysis** 
* InfoBatch outperforms the tuned baseline. This is properly because higher loss samples are 
more sensitive to learning rate scaling as discussed in Appendix B.3. 
* Based on that, when scaling the learning rate to match
the original training progress, scaling lower loss samples is still preferred than higher loss samples. 

[//]: # (We add this baseline to Table 1 in the revision.)

**Conclusions:**
* With the same computation, InfoBatch has higher performance than the tuned baseline.
* According to the table and the point above, the improvement of InfoBatch is not caused by a lack of tuning of the benchmark.

Reviewer comment (Theory):
Is there anything more to section 2.3 than showing unbiasedness?

Author response:
**Q5: Purpose of 2.3**

**A5:** Thanks for the question. Previously, considering the different backgrounds of reviewers, we wrote 2.3 with more detailed 
steps. We summarize it to make it brief (in the updated revision), the changes are as follows:

* We merge previous Eqn. (6) and (7) to revised Eqn. (6), and move the middle step to B.1.
* We merge previous Eqn. (8) and (9) to revised Eqn. (7).

Reviewer comment (Evaluation):
I would be curious what fraction of the data are soft-pruned throughout training? With the losses being bound below by 0, I would expect the distribution to become quite asymmetric as training proceeds. Could e.g. the median or some percentile be preferable? The median (not sure about arbitrary percentiles) can be computed in linear time, although I don't know if it is possible to update it online as for the mean.

Author response:
**Q7: What fraction of the data are soft-pruned throughout training?**

**A7:** Thanks for the question. We plot the saving fraction curve of InfoBatch in CIFAR-10 ResNet-50 training in 
revised Appendix E.1 Fig.6.

For **entropy loss**, InfoBatch (mean) usually **starts with a relatively lower ratio,
and then the ratio keeps increasing till the end**. This is probably because maximizing log probability tends to converge to a 
long-tailed loss distribution.

We show the loss distribution visualization during training in revised Appendix E.1 Fig.7. We can see the loss distribution
is initially right-skewed and finally left-skewed.

Reviewer comment (Experiments):
Do you have any thoughts on how to set \( r \) and \( \delta \) on a new dataset/architecture? The benchmarks in the paper are of course well-studied, but I think it would be a nice addition to the paper to discuss some heuristics for finding close-to-optimal values without needing a full training run (although suboptimal values already present a cost saving of course).

Author response:
**Q10: How to set r and $\delta$ on a new dataset/architecture?**

**A10:** Thanks for the good question. 

We consider the r value in the future work (see it in Appendix B.4), we plan to use 
$\mathbb{E}\_\text{rescaled}[G\_{z}^2/(1-\mathcal{P}\_t(z))]\leq {\mathbb{E}}\_{\mathcal{D}}[G_z^2]$
to find near-optimal r. We theoretically analyze the reasonableness of choosing r according to this equation.
* Our proposed procedure is to measure the $\mathop{\mathbb{E}}_{\mathcal{D}}[G_z^2]$ 
(or some cheaper indicator values instead) after warmup, then use 
the statistics to quickly select near optimal r values. 

For delta, our default value 0.875 is actually corresponding to the annealing stage of the one cycle scheduler 
we used for the learning rate schedule. It corresponds to a drop in loss during normal training. 
In general, it is the last 17.85% of a cosine annealing. As cosine annealing is prominent in 
many tasks, we suggest **last 0.1785 fraction of cosine annealing could be used as default**.

Reviewer comment (Evaluation):
InfoBatch improves over UCB via throwing away the dataset sorting operation. However, in Table 2, the practical overhead cost saving seems negligible compared to the wall clock time and the total node hour. Also, how was the overall saved cost calculated in Tables 6 and 7?

Author response:
**Q1: Overhead cost saving over UCB seems negligible compared to training**

**A1:** Thanks for the question. Sort is O(NlogN) for full data and amortized O(logN) for each sample, ours is O(N) and 
amortized O(1). We design the operation considering it could differ by a large factor when N is very big.

This overhead saving of operation compared to UCB is 10 times (0.03h to 0.0028h) on ImageNet-1K; however we agree it is 
not a significant improvement compared to training time saving (17.5h->10h). Our main saving comes from saving the forward and backward
computation, which is shown in the table below:


**Setting:** ResNet-50 on CIFAR-100. All results are averaged across three runs with std reported.
We report their acc and pruning ratio here. 

|           | Overhead (s) | Saving      | Acc. (%) |
|-----------|--------------|-------------|----------|
| UCB       | < 2          | 20%, ~800s  | 80.4±0.2 |
| UCB       | < 2          | 33%, ~1320s | 79.8±0.2 |
| UCB       | < 2          | 41%, ~1640s | 79.5±0.2 |
| InfoBatch | < 0.1        | 33%, ~1320s | 80.6±0.1 |
| InfoBatch | < 0.1        | 41%, ~1640s | 80.6±0.2 |

**Analysis**: 
* At lossless performance, InfoBatch can save 41% compared to 20% of UCB. The saved time is a 100% improvement.
* At the same saving ratio, UCB has a much higher performance degradation

We degraded the claim of the part on time complexity, which can be visible in the new revision. The main modifications can 
be summarized as:
1. in section 4 related works, we change "which could be a non-negligible overhead" -> "which could be an overhead"



**Q2: How was the overall saved cost calculated in Tables 6 and 7**

**A2:**
Thanks for the question. As the overhead is negligible compared to training, we measured wall clock time and found the sample iteration saving
is basically the same as wall clock saving. So the **overall saved cost is both wall clock time and sample iteration**.

Reviewer comment (Experiments):
To my understanding, annealing utilizes the whole dataset without pruning for the last 0.125% of total training epochs. This raises several concerns: (i) As the wall clock time of UCB and InfoBatch in Table 2 are both 10.5h, is this value taking the annealing process into account? (ii) Regarding Table 1, all the baselines and InfoBatch are compared under the same dataset pruning ratio. I wonder whether this is a fair comparison when annealing is involved in InfoBatch. (iii) Why did the authors utilize annealing only as a means of stabilizing the optimization, rather than leveraging the full dataset at the very beginning of optimization when we know that the early epochs of training can heavily influence the convergence of the loss landscape [1]?

Author response:
**Q3: Is annealing taken into account for saving?**

**A3:** Thanks for the question. The answer is yes. Our time = overhead + pruned training time + annealing training time.
For table 1, InfoBatch saves slightly more cost than reported, with annealing already taken into account.

**Q4: Using full dataset at warmup**

**A4:** Thanks for the question. Only in the experiment of Timm Swin-Tiny training, we warmup with the full dataset for 5 epochs to avoid training instability. 

On other tasks,the initial pruning ratio is lower, which serves as a pruning warmup for CNN-based experiments. A fraction
is visualized in revised Appendix E.1 Fig.6. It is properly due to early stage loss distribution is right-skewed and later left-skewed,
which is visualized in revised Appendix E.1 Fig.7.
ViT-based networks seem more sensitive to this warmup stage than CNN-based ones.

Thank you for pointing it out, we make the following modification to make it more clear:
* In sec A.3 paragraph 2, add "We also warmup InfoBatch for 5 epochs, only recording scores without pruning."

Reviewer comment (Evaluation):
The authors may need to provide further clarification regarding how annealing contributes to the stabilization of the rescaling process, especially if it does not seem to significantly impact the variance of the results in Table 4.

Author response:
**Q5: Further clarification of how annealing contributes to stabilization**

**A5:** Thanks for the question. We found Table 4 with a low number of precision digits is hard to illustrate this point clearly.
We draw a plot in the revised E.2 Fig.8. All experiments
use the same saving ratio.

**Analysis:** In this plot, compared to only rescaling, adding annealing leads to an increased mean performance and slightly reduced performance variance.

**Conclusion:** This plot indicates how annealing helps stabilize the training.

Reviewer comment (Evaluation):
The authors claim that the use of loss values in pruning conditions serves two purposes: (i) it reflects the learning status of samples, and (ii) it theoretically ensures unbiased gradient expectations. However, in Table 5, it is observed that even a random pruning criterion achieves nearly the same performance as the original pruning condition. This result raises questions about the necessity and effectiveness of using loss values as a pruning criterion and may require further discussion or clarification in the paper.

Author response:
**Q6: Necessity of using loss value**
**A6:** Thanks for the comment.

By reasonably utilizing loss, we can further improve data efficiency, based on the inference in Appendix B.3 (revised version B.4).

Noted that in Table 5 (CIFAR-100 R-50), the lossless pruning ratio using mean is 33%. In Section 3.1 experiment details
we claim to use a more aggressive pruning probability r(0.75) for smaller loss samples (lowest 20%) on CIFAR-100;
that leads to a 41.3% saving ratio in Table 6.

We conduct the following experiment to elaborate:

**Setting:** InfoBatch on CIFAR-100 ResNet-50 with increased r for certain samples. 
All results are averaged across three runs with std reported.
We report their acc and pruning ratio here.

| method                     | saving ratio (%) | performance (%) | time    |
|----------------------------|------------------|-----------------|---------|
| lower loss higher r(0.75)  | 41.3             | 80.6±0.2        | 1.48h   |
| higher loss higher r(0.75) | 41.3             | 79.9±0.3        | 1.48h   |
| random                     | 41.3             | 80.3±0.2        | 1.48h   |

**Analysis:** 
* Using a higher pruning ratio r for higher loss samples significantly degrades performance.
* A related discussion is in Appendix B.3 (revised version B.4) about why lower-loss samples are preferred. 
$\mathbb{E}\_\text{rescaled}[G\_{z}^2/(1-\mathcal{P}\_t(z))]\leq {\mathbb{E}}\_{\mathcal{D}}[G_z^2]$ could be 
the condition for a healthy rescaling.
* Compared to random selection, there is almost no overhead to use loss, as it can be obtained with no extra cost.

**Conclusions:** 
* **Loss is preferred, especially when using an adaptive high pruning ratio** which further improves data
efficiency. 
* The **overhead of loss is negligible**, so its **improvement is of almost no cost** and thus also preferred in settings 
without adaptive high pruning ratio.

Reviewer comment (Reproducibility):
How many random seeds are used throughout the experiments?

Author response:
**Q7: How many random seeds are used throughout the experiments**

A7: Thanks for the question. For CIFAR-10/100 experiments, we measured at least 3 trials for those values with std.

Reviewer comment (Novelty):
Missing recent works: 1) static data pruning [a,b,c], 2) dynamic data pruning [d]

Author response:
**A1:** Thanks for the advice. We update the section 1 and 4 to add those references(see the revision). The main changes are:
* Sec 1, paragraph 2: add reference " Park et al., 2022; Xia et al., 2023; Zheng et al., 2023"
* Sec 4, paragraph 1: add "AL (Park et al., 2022) propose
to use active learning methods to select a coreset. Moderate (Xia et al., 2023) proposed to use the
median of different scores as a less heuristic metric. Coverage-centric Coreset Selection (Zheng et al.,2023) 
additionally considers distribution coverage beyond sample importance. "
* Sec 4, paragraph 2: add "Mindermann et al. (2022) propose Reducible Holdout Loss Selection which
prioritizes samples neither too easy nor too hard. It emphasizes training learnable samples."

Reviewer comment (Presentation):
How to illustrate the gradient trajectory with landscape in Fig 1? Is it an illustration or a real plot on some dataset?

Author response:
**A2:** Thanks for the question. Fig 1 is an illustration instead of a real plot. To avoid misunderstanding, we modify the caption 
"visualiztion" -> "illustration".
The corresponding accuracy values are from Table 1 CIFAR100 experiment, comparing InfoBatch and EL2N-2 at 50% pruning ratio. 

**Improvement Plan:**
We notice that there could be a potential improvement of this illustration using the method proposed in [1].
It would express the same idea but with a more accurate visualization. We found in [2] Fig 1 they show the gradient sketch
on such a loss landscape. We are welcome to further discussion on this in the auther reviewer discussion period (whether change our Fig 1 like that).

[1] Li, Hao, et al. "Visualizing the loss landscape of neural nets." Advances in neural information processing systems 31 (2018).
[2] Liu, Zhuang, et al. "Dropout Reduces Underfitting." arXiv preprint arXiv:2303.01500 (2023).