Paper: Candidate Label Set Pruning: A Data-centric Perspective for Deep Partial-label Learning (ICLR-2024)

Reviewer comment (Writing):
A small typo? I guess the author may forget to divide n in the formulation of \beta in Definition 1, as \beta does not equal to 1 but n in the optimal pruning case. Can author clarify that?

Author response:
Thanks for pointing out this issue. We have corrected the issue in the formulation of $\beta$ to improve the clarity: $\beta=\frac{\sum _{i=1} ^{n}|\widetilde{Y} _{i}|}{\sum _{i=1} ^{n}(|Y _{i}|-1)}$. In the optimal pruning case, $\beta=1$ satisfies our consideration.

Reviewer comment (Writing):
It is vague to describe PLL in the abstract that "Partial-label learning (PLL) allows each training example to be equipped with a set of candidate labels." Does the author intentionally ignore the assumption that "only one is the ground-truth label"? It is better to clarify as some PLL research wave the limitation to investigate a new PLL task (called Unreliable or Noisy PLL).

Author response:
Thanks for pointing out this issue. We would like to explain that this assumption was formally introduced in the definition of PLL in Section 3.1 where each candidate label set $Y_{i}$ consists of a true label $y_{i}$ and a false candidate label set $Y_{i}^{\prime}=Y_{i}\backslash \{y_{i}\}$. To further improve the clarity of our paper, we have highlighted this assumption both in the abstract and problem setup in our paper.

Reviewer comment (Novelty):
Some state-of-art PLL methods are missing in the reference, such as, A Unifying Probabilistic Framework for Partially Labeled Data Learning; Mutual Partial Label Learning with Competitive Label Noise.

Author response:
Thank you for letting us know about the related work! We have introduced them [1, 2] in the related work section of our paper.  

**References:**  

[1] A Unifying Probabilistic Framework for Partially Labeled Data Learning. TPAMI. 2022.  

[2] Mutual Partial Label Learning with Competitive Label Noise. ICLR. 2023.

Reviewer comment (Reproducibility):
The proposed method is dependent on the KNN algorithm, which should be given more details in the main body of the paper. For example, it could be found in the appendix that the KNN algorithm are implemented on the output of a feature extractor. However, what the feature extractor comes from is unknown.

Author response:
**A:** Thanks for pointing out this issue. We have provided more details about the $k$-NN algorithm and the used feature extractors in Appendix C of our paper. Specifically, we employed the visual encoder of BLIP-2 [1] to extract 768-dimensional high-quality representations for all training instances in each PLL dataset. Then, we leveraged FAISS [2] to conduct the fast $k$-NN searching for each PLL instance based on the squared Euclidean Distance.

Reviewer comment (Writing):
What is the main difference between a data-centric method and a pre-processing method?

Author response:
**A:** Thanks for the interesting question. We think that our proposed data-centric method can be regarded as a special type of pre-processing method. This is because the proposed method can be used to pre-process training PLL instances before using these data to train deep PLL methods. We expect that our first data-centric method will motivate more studies on pre-processing methods for PLL.

Reviewer comment (Reproducibility):
Does the feature extractor in the KNN algorithm come from the classifier during the training process? If yes, it seems unreasonable to say that the method is training-free. If not, a pre-trained model should be introduced.

Author response:
**A:** Thanks for raising this concern. We would like to explain that the feature extractor does not come from the classifier during the training process. Instead, the feature extract comes from a multi-modal pre-trained model (e.g., CLIP, ALBEF, BLIP-2), which has not been trained on the corresponding PLL dataset. Hence, the proposed method is indeed training-free by leveraging a multi-modal pre-trained model. We have already introduced the pre-trained models in Section 4.1 of our original paper.
Notably, to evaluate the effect of different types of feature extractors, we additionally trained two ResNet-18-based models (i.e., ResNet-SSL, ResNet-S) on the corresponding PLL dataset. Note that we did not use them in the proposed method and just used them for comparing the effects of different types of feature extractors. Specifically, ResNet-S was trained on each PLL dataset with the original clean supervision, while ResNet-SSL was trained on each PLL dataset by the self-supervised learning method SimCLR [4] without any supervision. More details about the training setup of these two feature extractors have been shown in Appendix C of our paper.

Reviewer comment (Experiments):
The PASCAL VOC dataset used in the experiment is not introduced well, as the dataset is not originally for partial label learning.

Author response:
Thanks for raising the concern on the used PASCAL VOC dataset. In particular, we followed the previous work [1] to construct the dataset where objects in images were cropped as instances and all objects appearing in the same original image were regarded as the labels of a candidate set. In this way, we could obtain a PLL version of the PASCAL VOC dataset and perform any PLL method on this dataset. More characteristics of PASCAL VOC have been shown in Appendix C of our paper.

Reviewer comment (Presentation):
Detail of trained feature extractors (ResNet-SSL, ResNet-S) is shown unclearly.

Author response:
Thanks for pointing out this issue. Specifically, ResNet-S was trained on each PLL dataset with the original clean supervision using the cross-entropy loss, while ResNet-SSL was trained on each PLL dataset by the self-supervised learning method SimCLR [2] without supervision. We have further detailed the training setup of feature extractors in Appendix C of our paper.

Reviewer comment (Writing):
It would be appreciated if a clearer explanation for Definition 2 is provided. Why is this Definition needed?

Author response:
Particularly, Definition 2 aims to characterize the candidate label distribution in the local representation space. Specifically, a PLL instance’s $k$-NN instances are expected to have the true label in their candidate label sets with a high probability, while their candidate label sets are unlikely to have the same false candidate label. This characteristic contributes to the label distinguishability of candidate label sets in the local representation space, which is important for the proposed method to achieve a satisfying pruning error.

Reviewer comment (Reproducibility):
How are these values of k and $\gamma_{i}$ calculated in Figure 1?

Author response:
Thanks for pointing out this issue. We have improved the clarity of this part in Section 3.3 of our paper. Our objective of the numerical simulation experiment is to evaluate the effects of representations of different qualities and different label ambiguities on the upper bound of the pruning error, for the proposed method. For this purpose, we empirically set various values of $\delta_{k}$ and $\rho_{k}$ where a small value of $\delta_{k}$ ($\rho_{k}$) implies high-quality representations (candidate label sets of low label ambiguity). By substituting the specific values of $\delta_{k}$ and $\rho_{k}$, we can calculate an exact upper bound with varying values of $k$ and different numbers of eliminated candidate labels $\gamma_{i}$. In this way, we are able to know how to select suitable values of $k$ and $\gamma_{i}$, with different feature extractors and PLL settings in the experiments.

Reviewer comment (Evaluation):
Why does the loss curve on VOC in Figure 5 have a rise (except PRODEN algorithm)? This phenomenon is different from other cases on CIFAR and Tiny-ImageNet datasets.

Author response:
The phenomenon is true that the training loss curves of certain PLL methods have a rise after training with pruned PLL data on PASCAL VOC in Figure 5. We reckon that there are two reasons. Firstly, there are more proportional noisy PLL instances in PASCAL VOC (the pruning error is 5.2\%) than in CIFAR and Tiny-ImageNet (the pruning error is almost < 1\%). Secondly, label disambiguation towards PLL instances in PASCAL VOC is more challenging due to the complicated visual objects in PASCAL VOC. Hence, fitting noisy PLL instances in PASCAL VOC is more difficult, thereby leading to a larger training loss value. More details are shown in Appendix D of our paper.

Reviewer comment (Experiments):
How is the PASCAL VOC dataset used for partial label learning algorithms?

Author response:
Thanks for raising the concern on the used PASCAL VOC dataset. In particular, we followed the previous work [1] to construct the dataset where objects in images were cropped as instances and all objects appearing in the same original image were regarded as the labels of a candidate set. In this way, we could obtain a PLL version of the PASCAL VOC dataset and perform any PLL method on this dataset. More characteristics of PASCAL VOC have been shown in Appendix C of our paper.

Reviewer comment (Evaluation):
What is the potential limitation of the proposed approach? Discussing this point is also important to have a comprehensive understanding for the proposed approach.

Author response:
Thanks for raising the concern. We think that a potential limitation of the proposed method is the presence of noisy PLL instances in the pruned PLL dataset whose true label is incorrectly eliminated and inside the non-candidate label set. Besides, many PLL methods are incapable of dealing with noisy PLL instances in the pruned PLL datasets. To alleviate this issue, as analyzed in the numerical simulation experiment, we can decrease the upper bound of the pruning error by leveraging high-quality representations extracted from advanced feature extractors and using appropriate parameters in the proposed method, thereby effectively reducing the number of noisy PLL instances in the pruned PLL dataset. In addition, it would be also interesting to deal with potential noisy PLL instances directly during the pruning procedure.

Reviewer comment (Experiments):
More empirical analysis in the experiment should be presented, such as which PLL algorithms are more sensitive to the pruning method.

Author response:
**A:** Thanks for your suggestion! We have provided additional analyses on the experimental results of the transductive accuracy comparison in Table 7 and analyses on the training loss curves in Figure 5. For the transductive accuracy comparison in Table 7, we discover that naive PLL methods (e.g., CC, PRODEN, LWS, and CAVL) have more significant performance improvements than advanced PLL methods (e.g., PiCO, CRDPLL, ABLE, and IDGP). We think that this is because naive PLL methods have a limited capability in label disambiguation compared with advanced PLL methods, hence the proposed CLSP method has a bigger effect on promoting label disambiguation in naive PLL methods. For the training loss curves in Figure 5, we found an unusual phenomenon where the training loss values of many PLL methods become larger on PASCAL VOC after training with pruned PLL data. We reckon that there are two reasons. Firstly, there exists a higher proportion of noisy PLL instances in PASCAL VOC (where the pruning error is about 5.2\%) than in CIFAR and Tiny-ImageNet (where the pruning error is less than 1\% in most cases). Secondly, label disambiguation of training PLL instances in PASCAL VOC is more challenging due to the complicated visual objects in PASCAL VOC. Hence, fitting noisy PLL instances in PASCAL VOC is more difficult, thereby leading to a larger training loss value. More details of these empirical analyses have been shown in Appendix D of our revised paper.

Reviewer comment (Evaluation):
More explanations on bad cases are needed to show the limitation of the proposed method.

Author response:
**A:** Thanks for raising the concern about bad cases! We have explained the potential reason for the based cases in the experiment results of our paper. Specifically, there are a few bad cases (4/149 $\approx$2.7\%) in the experimental results where the performance of certain PLL methods has slightly degraded after training with pruned PLL data. We argue that this is because the involved PLL methods (i.e., ABLE and SoLar) in the bad cases have a time-consuming training procedure (500 and 1000 epochs respectively), hence they tend to overfit noisy PLL instances eventually, thereby leading to performance degradation.

Reviewer comment (Experiments):
How about pruning on noisy PLL data? I am curious about whether the proposed method could be used on noisy PLL data whose true label is outside the candidate label set.

Author response:
**A:** Thank you for this interesting point! Regrettably, the proposed CLSP method may be infeasible to handle noisy PLL instances directly. This is because our method only focuses on the candidate label set and thus has no effects on noisy PLL instances whose true label is inside the non-candidate label set.  Fortunately, when the number of noisy PLL instances in the original dataset is not large, the proposed CLSP method can be still applied.

Reviewer comment (Evaluation):
I find some bad cases in the experiment (shown in Table 1 and Table 8) where the performance drops after pruning. How to explain this phenomenon?

Author response:
**A:** Thanks for raising the concern about bad cases! We have explained the potential reason for the based cases in the experiment results of our paper. Specifically, there are a few bad cases (4/149 $\approx$2.7\%) in the experimental results where the performance of certain PLL methods has slightly degraded after training with pruned PLL data. We argue that this is because the involved PLL methods (i.e., ABLE and SoLar) in the bad cases have a time-consuming training procedure (500 and 1000 epochs respectively), hence they tend to overfit noisy PLL instances eventually, thereby leading to performance degradation.

Reviewer comment (Presentation):
Could you show the values of delta_k and pho_k on the real-world dataset VOC (which are not shown in Figure 3)?

Author response:
**A:** Thanks for your suggestion. Following the empirical formulation of $\delta_{k}$ and $\rho_{k}$ shown in Section 4.1, we have conducted additional experiments to calculate their empirical values on PASCAL VOC. The calculated values of (1-$\delta_{k}$) and $\rho_{k}$ are shown in the following tables. 

|$k$|5|20|50|100|150|200|
|:--: |:--: |:--: |:--: |:--: |:--: |:--: |
|1-$\delta_{k}$|.792|.766|.736|.700|.676|.550|

|$k$|5|20|50|100|150|200|
|:--: |:--: |:--: |:--: |:--: |:--: |:--: |
|$\rho_{k}$|.655|.604|.580|.565|.557|.550|

From the above tables, we can see that the values of (1-$\delta_{k}$) and $\rho_{k}$ both decrease progressively as the value of $k$ increases. Further compared with the values on CIFAR in Figure 3, the values of (1-$\delta_{k}$) on PASCAL VOC are smaller than on CIFAR, which implies the quality of representations on PASCAL VOC is lower than on CIFAR. On the other hand, the values of $\rho_{k}$ on PASCAL VOC are larger than on CIFAR in most cases, indicating that label ambiguity in PASCAL VOC is more severe than in CIFAR. Hence, the proposed CLSP method is more challenging to employ on PASCAL VOC.

Reviewer comment (Experiments):
A unified proportion tau for each training instance is used in Eq. (2). Are there other ways to adaptively control the number of eliminated candidate labels for each training instance?

Author response:
**A:** The answer is positive! It is feasible to eliminate different numbers of candidate labels in training PLL instances in an adaptive manner. We can formulate the problem of adaptively deciding the optimal number of eliminated candidate labels for each training PLL instance as an integer programming problem. In this case, certain optimization techniques might be required to solve the integer programming problem.