Paper: BOSS: Diversity-Difficulty Balanced One-Shot Subset Selection for Data-Efficient Deep Learning (ICLR-2024)

Reviewer comment (Theory):
Rather than rigorously derived from the balanced core-set loss bound, equation (5) seems to be simply a hand-crafted heuristic selection strategy combining the diversity-based method and the difficulty-based method. In theroem2, the authors claim that EL2N lower bound the label variability in difficult regions. I wonder if this holds for other regions as EL2N/difficulty is universally used in Equation (5). Besides, to minimize Equation (1), for the label variability term, we should minimize something upper bounds $|| \boldsymbol{y}_i -  \boldsymbol{y}_j ||$ instead of something lower bounds it like EL2N.

Author response:
**Q1: Rather than rigorously derived from the balanced core-set loss bound, Equation (5) seems to be simply a hand-crafted heuristic selection strategy combining the diversity-based method and the difficulty-based method**

We would like to clarify that Equation 5 closely follows our theoretical result in Theorem 1 that decomposes the overall loss bound into the input similarity and label variability. As mentioned in our general response, the involvement of the label variability makes our method substantially different from existing methods (e.g., those based on feature/gradient similarity or the standard facility location function as commented by other reviewers). Another major novelty also comes from the importance function $I(\cdot)$ that goes beyond a difficulty based metric. It has a much richer functional form that connects the label variability (difficulty) and subset size. This allows us to achieve a dynamic balance between the input similarity and label variability based upon the subset size.  The importance function also bridges the theoretical objective of loss bound given in Theorem 1 and the empirical objective of submodular maximization of Equation 5. It is worth to note that this connection is essential because Equation 1 cannot be directly implemented as the model property ($\lambda^{\boldsymbol{\eta}}$) is unknown.  

**In theorem 2, the authors claim that EL2N lower bound the label variability in difficult regions. I wonder if this holds for other regions as EL2N/difficulty is universally used in Equation (5)** 

The *label variability* is mostly important when we want to select samples from difficult regions or points near the decision boundary. When we move towards the easier region both the EL2N score and label variability are significantly lower compared to the difficult region (as verified in the visualization of synthetic datasets).  So among the easier samples, we only focus on selecting representative samples, and the values of label variability will be small anyway. 

**Q2: Besides, to minimize Equation (1), for the label variability term, we should minimize something upper bounds $||{\bf y}_i-{\bf y}_j||$ instead of something lower bounds it like EL2N** 

This is a great question! Please refer to our general response regarding "Connecting label variability with EL2N score".

Reviewer comment (Theory):
The authors claim that the subset size will affect the optimal diversity-difficulty balance, in data data-scarce regime, the diversity dominates while as the subset budget increases, more difficult samples should be picked. While intuitively true and the authors give intuitive explanations, I can’t directly justify the statement directly from the core-set loss bound. More discussion will greatly strengthen the paper.

Author response:
**Q3: The authors claim that the subset size will affect the optimal diversity-difficulty balance, in data data-scarce regime, the diversity dominates while as the subset budget increases, more difficult samples should be picked. While intuitively true and the authors give intuitive explanations, I can’t directly justify the statement directly from the core-set loss bound** 

Thank you for this insightful question! To more clearly show how the subset size impacts the balanced core-set loss bound, we conduct additional experiments on the synthetic data, aiming to quantify and visualize the two major components in the loss bound: $\sum_i||{\bf x}_i-{\bf x}_j||$ and $\sum_i||{\bf y}_i-{\bf y}_j||$, which essentially captures the feature distance and label distance between the selected subset and the full set, respectively. As can be seen from Figure 9 (a) in the Appendix of the revised paper, for a small subset size, when choosing the subset based on the label variability (or difficulty), it can help to quickly reduce the label distance. However,  it also leads to a very large feature distance that makes the overall bound large. Figure 9 (b) further confirms this because the selected samples misses some major regions of the data distribution as stated in the main paper. In contrast, when focusing on choosing samples based on the first component (i.e., diversity), the feature distance drops significantly as shown in Figure 9 (c), which implies that the selected subset can represent the entire data distribution well. This is further confirmed by  Figure 9 (d), which visualizes the distribution of the selected data samples. As more samples are selected, they will start to cover the difficult regions, which can effectively bring down the label distance as shown in Figure 9 (c).

Reviewer comment (Experiments):
For a fixed number of epochs, the entire dataset must be used to train a model.

Author response:
**Q1: the entire dataset must be used to train a model for a fixed number of epochs** 

We would like to clarify that it is a common practice for one-shot subset selection methods to perform initial training on full data for the first few epochs. This is followed by all the major one-shot subset methods, including CCS, LCMAT, Moderate, EL2N, and Forgetting. Such initial full-set training is essential to obtain necessary information (e.g., gradients, features, EL2N scores) about the data before subset selection can be performed. Furthermore, since the initial training only takes a few epochs, it introduces limited overhead (see our response to Q3 for details). 

When compared with the dynamic subset selection, although a small number of samples is used to train the model at a time, the dynamic subset selection will be conducted using a large fraction of data throughout the training. This is not feasible in applications such as continual learning where the full data is accessible only once.

Reviewer comment (Experiments):
Absence of variety in experiments. ResNet is insufficient on its own to verify the efficacy of the proposed method. To validate their approach, it is necessary to conduct experiments on more models.

Author response:
**Q2: experiments on more models** 

Thank you for this great suggestion! We have conducted additional experiments on both EfficientNet [1] and ViT [2]. Overall, we have used four models ResNet18 (for SVHN, CIFAR10, and CIFAR100), ResNet34 (for TinyImageNet), EfficientNet, and ViT (for CIAFR100). We are able to show the effectiveness and consistent trend of our method as compared with competitive baselines for all of these models. 

[1] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6105-6114, 2019.

[2] Dosovitskiy, Alexey, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." International Conference on Learning Representations. 2021.

ViT Result (CIFAR100):
| Subset | Random | Moderate | CCS | BOSS |
|--------|--------|----------|-----|------|
| 10%    | 78.49 ± 0.7| 50.41 ± 0.7| 78.62 ± 0.3| **79.97 ± 0.4**|
| 20%    | 81.87 ± 0.7| 69.81 ± 0.5| 81.95 ± 0.8| **83.19 ± 0.1**|
| 30%    | 83.98 ± 0.2| 77.66 ± 0.5| 84.93 ± 0.1| **85.08 ± 0.1**|
| 50%    | 85.88 ± 0.1| 84.19 ± 0.0| 85.88 ± 0.1| **86.55 ± 0.1**|

Efficient Net Result (CIFAR100):
| Subset | Random | Moderate | CCS | BOSS |
|--------|--------|----------|-----|------|
| 10%    | 30.51 ± 1.0| 32.59 ± 1.3| 36.91 ± 2.2| **42.64 ± 0.6**|
| 20%    | 43.52 ± 1.9| 42.04 ± 2.2| 46.53 ± 3.7| **53.39 ± 0.3**|
| 30%    | 55.48 ± 0.7| 55.26 ± 1.7| 56.89 ± 0.3| **60.37 ± 0.4**|
| 50%    | 64.05 ± 0.7| 63.91 ± 0.3| 63.59 ± 0.5| **68.27 ± 0.5**|

Reviewer comment (Experiments):
There is no comparison between the entire train duration and the time required to generate a subset. The problem with the proposed process is that all of the data must be trained so that authors should perform experiments with computation complexity.

Author response:
**Q3: comparison between train time and selection time; experiments with computation complexity** 

We compare the time taken for *Subset Selection*, and *Subset Training*. The *Subset Selection* consists of initial training for 10 epochs on the full set, and the lazy greedy algorithm to select the subset. The *Subset Selection* time is shorter compared to training on the subset (*Subset Training*) and is much more efficient compared to training on the full set (*Full Set Training*). The time for the subset selection algorithm (time excluding the initial training) is significantly small compared to the initial training time and this step does not require GPU computation. 
We measure the time in seconds using NVIDIA RTX A6000 GPU for CIFAR10 and CIFAR100 datasets.

Time comparison results
| Dataset   | Subset Size | Subset Selection (Initial Training) | Subset Selection (Selection Algorithm) | Subset Training | Full Set Training |
|-----------|-------------|-------------------------------------|-----------------------------------------|------------------|-------------------|
| CIFAR100  | 10%         |                                     | 9                                   | 346              |                   |
|           | 20%         | 219                                 | 13                                  | 587              | 4387              |
|           | 30%         |                                     | 14                                  | 800              |                   |
|           | 50%         |                                     | 15                                  | 1816             |                   |
| CIFAR10   | 10%         |                                     | 11                                  | 342              |                   |
|           | 20%         | 173                                 | 19                                  | 571              | 3468              |
|           | 30%         |                                     | 22                                  | 801              |                   |
|           | 50%         |                                     | 27                                  | 1244             |                |

Reviewer comment (Writing):
It is difficult to discern what the author intended when they write, "missed some critical regions(upper middle area)", as Figure 1(a) on page 2.

Author response:
**Q4: meaning of "missed some critical regions", Figure 1(a)** 

The critical regions refer to the regions near the boundary. We have annotated those regions in Figure 1 on page 2 of the revised paper.

Reviewer comment (Writing):
What does the symbol gamma represent in Theorem.1 on page 4?

Author response:
**Q5: what does $\gamma$ represent in Theorem 1?** 

This term is related to Hoeffding's bound. $\gamma$ is the probability that the bound does not hold true.  $1 - \gamma$ is the probability that the bound holds true.

Reviewer comment (Theory):
What is the rationale behind the paper's assertion that "CCS still does not strike the right balance between diversity and the difficulty of subset selection"?

Author response:
**Q6: rationale behind "CCS still does not strike the right balance between diversity and difficulty"** 

As we have shown with the theoretical analysis (please see our general response for more details), the optimal selection requires targeting different difficulty levels according to the subset size, which is impossible for CCS.

Reviewer comment (Experiments):
The proposed function is very similar to the standard facility location function, which is $\sum_{i \in V} max_{j \in A} Sim(x_i, x_j).$ The function additionally has the I(.) term which is the main contribution in my opinion. To fully understand the effect of the additional I(.) term, the authors should compare with the facility location submodular function.

Author response:
Thank you for the suggestion. In the ablation study, we present subset selection using only the diversity component, which is the same as the standard submodular facility location function as suggested by the reviewer. Our result shows that by maximizing the combination of the *standard facility location function* (diversity) and the *importance function* (function of difficulty and subset size), we can achieve significant improvement compared to only using the *standard facility location submodular function* (diversity).  

Additionally, we want to emphasize that the importance function $I(\cdot)$ itself is not the only main contribution of our work. Our contribution also lies in the theoretical work bringing together the input similarity, label variability, and subset size, which justifies the need to maximize the joint objective of the importance function and submodular function (Equation 5) in order to minimize the loss bound in Equation 1.

We also compare with other baselines such as CRAIG and Adacore which are also based on the submodular location function but use the gradients instead of the features.

Reviewer comment (Experiments):
The authors discuss multiple relevant papers in this work but do not add comparison with many of them in the experiments. It would be great to compare with a few more method, e.g., Grad Match.

Author response:
Following the reviewer's suggestion, we have compared with two additional baselines, including GradMatch and Adacore. The complete table is updated in the revised paper (see Table 1 on page 8). Our proposed method shows a clear advantage over these two baselines on all settings, which further justifies its effectiveness. We would also like to clarify that the representative-based subset selection methods are not initially used for one-shot subset selection but rather for dynamic subset selection. Therefore, the performance cannot be directly compared to the respective papers as we are evaluating them in a one-shot setting. 

| Dataset       | Subset | GradMatch | Adacore | BOSS(Ours) |
|---------------|--------|-----------|---------|------------|
| Tiny ImageNet | 10%    | 23.68 ± 1.5| 24.12 ± 1.5| **33.22 ± 0.5**|
|               | 20%    | 38.20 ± 1.3| 37.94 ± 0.6| **45.73 ± 0.4**|
|               | 30%    | 44.93 ± 0.6| 44.72 ± 0.5| **51.75 ± 0.4**|
|               | 50%    | 53.81 ± 0.2| 53.37 ± 0.4| **57.88 ± 0.2**|
| CIFAR 100     | 10%    | 36.68 ± 0.6| 37.65 ± 0.8| **47.58 ± 0.5**|
|               | 20%    | 53.16 ± 2.2| 52.79 ± 0.8| **61.44 ± 0.7**|
|               | 30%    | 63.02 ± 1.2| 62.28 ± 1.2| **67.89 ± 0.2**|
|               | 50%    | 70.68 ± 0.4| 71.19 ± 0.3| **74.03 ± 0.3**|
| CIFAR 10      | 10%    | 72.26 ± 0.5| 72.65 ± 0.9| **79.47 ± 0.5**|
|               | 20%    | 84.30 ± 0.9| 84.30 ± 1.2| **87.82 ± 0.9**|
|               | 30%    | 88.47 ± 0.6| 88.37 ± 1.2| **92.15 ± 0.6**|
|               | 50%    | 91.89 ± 0.4| 92.67 ± 0.5| **94.36 ± 0.2**|
| SVHN          | 8%     | 84.31 ± 1.8| 82.31 ± 2.6| **89.52 ± 0.8**|
|               | 12%    | 88.99 ± 1.0| 88.41 ± 1.3| **93.18 ± 0.5**|
|               | 16%    | 90.42 ± 0.8| 90.34 ± 0.8| **94.31 ± 0.3**|
|               | 20%    | 91.56 ± 0.4| 91.95 ± 0.8| **95.08 ± 0.3**|

Reviewer comment (Experiments):
The 'balanced' aspect of the proposed loss is still not clear to me. It would be imperative to add some experiments to show how the selected subsets are balanced. It would be even better if the authors can show some experiments on class imbalanced data. Most datasets currently in the experiments barely have any imbalance, which makes this analysis difficult.

Author response:
We would like to clarify that the 'balanced' keyword in our work refers to the interaction of the *input similarity* component and the *label variability* component of the loss bound as derived in Theorem 1 with respect to the subset. Specifically, focusing on improving one component might make another component worse thus there is a need for balancing between those two components. 

In our evaluation, we follow the standard setting of most existing works, where the methods are primarily evaluated on balanced data. However, we understand that in a real-life scenario, the data could be imbalanced. So we can certainly evaluate our method in some class imbalance setting.

We have employed two ways of creating imbalanced data: exponential and step-wise. In exponential, we decrease the class size using exponential decay $N_{c_i}\times e^{-0.01i}$ where $N_{c_{i}}$ is the number of samples for class $c_i$. In step-wise imbalance, we prune 80\% of data from 20\% of classes. For the baselines, we chose the most recent competitive methods (CCS and Moderate) in addition to the random method. As can be seen, the results show our method consistently outperforms the competitive baselines under different class imbalance settings. 

**To be continued...**

**Continuing...**

Imbalance result for CIFAR 100, exponential
| Subset | Random | Moderate | CCS | BOSS |
|--------|--------|----------|-----|------|
| 10%    | 27.39 ± 0.9| 25.37 ± 1.7| 29.41 ± 0.5| **36.63 ± 0.9**|
| 20%    | 42.82 ± 1.1| 40.57 ± 0.6| 44.36 ± 1.7| **50.91 ± 0.6**|
| 30%    | 52.51 ± 0.7| 50.00 ± 3.0| 50.87 ± 1.4| **57.15 ± 0.5**|
| 50%    | 63.03 ± 0.6| 61.76 ± 0.2| 61.86 ± 0.5| **66.72 ± 0.1**|

Imbalance result for CIFAR 100, step-wise
| Subset | Random | Moderate | CCS | BOSS |
|--------|--------|----------|-----|------|
| 10%    | 31.66 ± 0.7| 27.02 ± 0.7| 33.60 ± 0.9| **38.72 ± 0.9**|
| 20%    | 47.36 ± 0.9| 41.77 ± 2.9| 46.82 ± 0.6| **53.75 ± 0.3**|
| 30%    | 56.64 ± 0.2| 52.31 ± 0.6| 52.81 ± 1.1| **58.37 ± 0.4**|
| 50%    | 62.19 ± 0.9| 59.96 ± 0.7| 60.25 ± 0.2| **66.03 ± 0.2**|