Paper: BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges (ICLR-2024)

Reviewer comment (Theory):
Using kernel ridge regression to decide the best window is not quite intuitive. What is the motivation to use kernel ridge regression rather than training a small network to decide the best window?

Author response:
We appreciate the reviewer for addressing this important question. Our use of the kernel ridge regression as a proxy for training neural networks can be partly explained by the recent progress in theoretical understanding of training neural networks using kernel methods. In particular, some recent works [a,b,c,d] have shown that training and generalization of neural networks can be approximated by two associated kernel matrices: the Conjugate Kernel (CK) and Neural Tangent Kernel (NTK). The Conjugate Kernel is defined by the gram matrix of the derived features produced by the final hidden layer of the network, while NTK is the gram matrix of the Jacobian of in-sample predictions with respect to the network weights. These two kernels also have fundamental relations in terms of their eigenvalue distributions as analyzed in [e]. Our proxy task is motivated by observation that the kernel regression with these model-related kernels can provide a good approximation to the original model (under some assumptions such as enough width, random initialization, and small enough learning rate, etc.).

In particular, we use the Conjugate Kernel (CK) in our kernel ridge regression (Equation (2)), by defining the kernel matrix as ${\mathbf{X}\_\mathbf{S}}^\top {\mathbf{X}\_\mathbf{S}}$ where $\mathbf{X}\_\mathbf{S}=[\mathbf{f}\_1,\dots,\mathbf{f}\_m]$ is composed of features produced by the exact target network of our consideration (ResNet18 for CIFAR-10 and ResNet50 for CIFAR-100/ImageNet). By considering the features from the target network, we can obtain the (approximate) network predictions that are linear in these derived features. Of course, this kernel approximation of the neural network models, which assumes a fixed feature extractor, does not exactly match our situation where the selected subset not only affects the linear classifier but also the feature extractor itself during the training. However, we’d like to emphasize that this is a proxy that can reflect the network architecture of our interest in a computationally-efficient manner. Also, our analysis in Table 2 shows that this proxy finds the best window subset that aligns well with the result from the actual training of the full model.

[a] Radford M Neal. Bayesian learning for neural networks, 1995.

[b] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes, ICLR 2018.

[c] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net, NeurIPS 2019. 

[d] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018.

[e] Zhou Fan and Zhichao Wang. Spectra of the Conjugate Kernel and Neural Tangent Kernel for Linear-Width Neural Networks, NeruIPS 2020.

Reviewer comment (Evaluation):
The baseline evaluation results are inconsistent with data reported in the baseline method. For example, moderate are reported to have better performance than random on CIFAR10. CCS seems to have a better performance at 10% subset ratio than the numbers reported in the paper. It may be good to explain why the difference exists.

Author response:
The differences between our CCS [f] numbers and those reported in the original paper stem from variations in the hyperparameter, $\beta$, employed within the algorithm. CCS prunes a $\beta$\% of hard examples, with $\beta$ being a hyperparameter, and then selects samples with a uniform difficulty score distribution. Considering $\beta$ as a one-sided threshold for the window subset, tuning this parameter can be viewed as an oracle window search within our algorithm. Thus, for a fair comparison we set beta to 0 for all the pruning ratios, and this makes our evaluation of CCS the same as the 'stratified only' algorithm in Table 2 of the CCS paper. When we set beta to 0, it operates optimally when the subset ratio is high but sub-optimally when the ratio is low. This observation also aligns with the findings in the CCS paper Fig 6(b). Below are the comparisons between the reported numbers for CCS in CIFAR-10 dataset.

| Subset ratio                              | 10%   | 20%   | 30%   | 50%   |
|:-----------------------------------------:|:-----:|:-----:|:-----:|:-----:|
| CCS in original paper                     | 85.7  | 90.93 | 92.97 | 95.04 |
| Stratified only(beta 0) in original paper | 59.23 | 81.82 | 90.78 | 95.13 |
| CCS in Ours                               | 81.56 | 89.28 | 92.5  | 94.78 |

For the moderate coreset [g], the results for CIFAR10 were not provided in the original paper, so we could not directly compare the numbers. In the case of CIFAR100, we reported better results as summarized in the table below. This discrepancy arises from differences in experimental settings; we preserve the number of iterations per epoch regardless of the subset ratio, whereas the original paper varies the iterations according to the subset ratio.


| Subset ratio                  | 20%   | 30%   | 40%   |
|:-----------------------------:|:-----:|:-----:|:-----:|
| Moderate DS in original paper | 51.83 | 57.79 | 64.92 |
| Moderate DS in Ours           | 58.01 | 63.97 | 68.51 |

[f] Coverage-centric Coreset Selection for High Pruning Rates. H Zheng et al.

[g] Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning. X Xia et al.

Reviewer comment (Theory):
The notion of a "window" refers to a fixed-length interval within a sorted dataset. The "Best Window Selection (BWS)" algorithm operates under the assumption that the most optimal subset should be contiguous regarding the level of difficulty. However, the paper lacks an in-depth analysis of this particular aspect.

Author response:
We’d like to highlight that the rationale behind the “window” selection from a sorted dataset mainly lies on two critical merits of this approach: 1) computational efficiency in searching for the optimal subset and 2) flexibility in choosing the data subset, enabling the selection of easy, moderate, or hard data subsets. In particular, this flexibility is pivotal in achieving competitive performances across a broad range of selection ratios.

As pointed out by the reviewer, the optimal subset might not be necessarily strictly contiguous in terms of the difficulty level of the samples. However, our theoretical analysis in Sec. 3.1 and the ablation study in Table 3 deliver some insights on the benefit of choosing samples from the continuous interval. First of all, in the simple toy example of binary classification problem introduced in Sec. 3.1, we show that the difficulty level of the optimal subset indeed changes depending on the subset size. In particular, for sample-deficient regimes, it is better to select easy samples (that are farther from the decision boundary), while selecting difficult samples (closer to the decision boundary) is more beneficial for sample-sufficient regimes. This theoretical analysis aligns well with the empirical findings (summarized in Table 1) that the previous score-based selection methods, which select rare or difficult-to-classify samples, achieve a better performance in high selection ratios, while the coreset selection methods, which select representative samples approximating the average behavior of the total dataset, achieves a stronger performance in low selection ratios. So, we can see that the optimal subset should be composed of relatively easy (hard) samples in low (high) selection ratios. 

Then, the remaining question is whether there exists such a desirable difficulty level for the subset in the intermediate selection ratios. We do not provide theoretical analysis on this aspect, but present some empirical evidence in our ablation study. In particular, our ablation study in Table 3 compares different types of subsets (in addition to the hard-only or easy-only windows), such as “Hard-easy”, which combines the easy and hard samples with equal proportions or “25-75\%”, which includes random samples from a moderate regime after pruning a fixed portion of too easy or hard samples. Compared to these non-contiguous subset selection methods, our window selection consistently achieves better performance over all the selection ratios (10 to 40\%) by cleverly choosing the window starting point. This ablation study, combined with our pruning experiment results summarized in Figure 4, partly demonstrate that our window selection, which prunes top $s$\% of the most difficult samples and top $(100-w-s)$\% of the easiest samples with a properly chosen $s$ for a fixed width of $w$\%, provides a good enough flexibility in choosing a subset of competitive performances across a broad range of selection ratios. 

Lastly, we provide further evidence that even though our method selects samples from a contiguous range of difficulty level, our subset maintains diversity in sample selection (in the embedding space), which is often considered as a desirable criteria in subset selection. To show this, we compute the coverage (\%) of the full dataset from the $k$-nearest neighbors of the selected samples from our method (BWS), and compare this coverage with that of random selection. For example, for the subset ratio of 10% and $k=3$, if the coverage is 30\%, it means that all the selected samples do not share their neighborhoods within distance 3 in the embedding space. The tables below show that the coverage of our method is as high as that of the random selection, showing that selecting samples of similar difficulty does not reduce the sample diversity in the embedding space compared to the random selection. 

The portion included in the 3-nearest neighborhood (k=3)
| Subset ratio  | 1%   | 5%    | 10%   | 20%   | 30%   |
|:-------------:|:----:|:-----:|:-----:|:-----:|:-----:|
| Window subset | 2.9% | 13.2% | 25.2% | 42.7% | 55.0% |
| Random subset | 2.9% | 13.9% | 26.0% | 45.3% | 59.6% |

The portion included in the 5-nearest neighborhood (k=5)
| Subset ratio  | 1%   | 5%    | 10%   | 20%   | 30%   |
|:-------------:|:----:|:-----:|:-----:|:-----:|:-----:|
| Window subset | 4.8% | 19.3% | 36.5% | 56.2% | 67.4% |
| Random subset | 4.8% | 21.7% | 38.3% | 60.7% | 74.2% |

Reviewer comment (Experiments):
It would be intriguing to explore the broader scenario where a "window" comprises several smaller intervals and varying starting points.

Author response:
We appreciate the reviewer for the interesting suggestion. We conducted an additional experiment on the CIFAR-10 dataset for finding the optimal two half-width windows while varying their starting points. In detail, for a subset of size $w$\%, we searched over all combinations of two half-width windows, denoted by $[x_1, x_1+w/2] \cup [x_2, x_2+w/2]$ while varying their starting points $(x_1, x_2)$ in $x_1\in[0, 100-w]$ and $x_2\in [x_1+w/2, 100-w/2]$ with a step size of 5\%. We trained ResNet18 on each subset and checked the corresponding test accuracies. Below we report the top five results (the compositions of half-width windows and their test accuracies) for subset ratios ranging from 10 to 40\%. We highlight the cases where the two half-width windows are contiguous to each other with bold letters. 

We can observe that for every considered subset ratio, the top-five best performing cases include contiguous windows (or windows near to each other with the gap of only 5\%), even though we allowed flexibility in choosing the two half-width windows far away from each other. This result further supports our claim (in the response to the reviewer’s first question) that our window selection, which chooses a continuous interval of samples based on difficulty scores, successfully finds the near-optimal subset in an efficient manner across a broad range of selection ratios. The related results are added in Appendix G.3 of our revised paper. 

| Subset ratio |  Rank             | 1st            | 2nd            | 3rd            | 4th            | 5th            |
|:--------------:|:---------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|
| 10%          | Half-width windows     | **35-40%, 40-45%** | **40-45%, 45-50%** | 30-35%, 45-50% | 40-45%, 50-55% | 40-45%, 55-60% |
|          | Test accuracy | **88.05**          | **87.82**         | 87.8           | 87.74          | 87.71          |
| 20%          | Half-width windows     | **20-30%, 30-40%** | 15-25%, 35-45% | 20-30%, 35-45% | **15-25%, 25-35%** | **25-35%, 35-45%** |
|          | Test accuracy | **91.69**         | 91.61          | 91.56          | **91.34**          | **91.29**          |
| 30%          | Half-width windows    | 10-25%, 30-45% | **10-25%, 25-40%** | 5-20%, 45-60%  | **15-30%, 30-45%** | 15-30%, 35-50% |
|          | Test accuracy | 93.47          | **93.35**          | 93.31          | **93.17**          | 93.11          |
| 40%          | Half-width windows     | 5-25%, 30-50%  | **5-25%, 25-45%**  | 0-20%, 25-45%  | 5-25%, 35-55%  | 5-25%, 40-60%  |
|           | Test accuracy | 94.68          | **94.38**          | 94.35          | 94.23          | 94.11          |

Reviewer comment (Presentation):
Figure 3's readability could be enhanced by employing more distinguishable colors and markers for clarity.

Author response:
We modified the figures (Figure 3, 4 and 6) by adding markers for better readability. Thank you for your feedback.