Paper: Can Class-Priors Help Single-Positive Multi-Label Learning? (ICLR-2024)

Reviewer comment (Writing):
The writing of this paper needs to be improved. Specifically, more analysis and descriptions of Theorem 4.2 are necessary.

Author response:
Theorem 4.2 provides an upper bound on the difference between the classifier obtained by the empirical risk minimizer $ \hat{f}_{sp} $ and the classifier corresponding to the true risk minimizer $ f^* $. This bound consists of four terms, which can be categorized into two groups: the Rademacher complexity terms and the terms of order $ O(1/\sqrt{n}) $.

   1. **Terms of Order $ O(1/\sqrt{n}) $**: As the sample size $ n $ grows, these terms reduce, indicating that $ \hat{f}_{sp} $ converges to the performance of $ f^* $ within a margin of error that becomes progressively smaller with more data.
   2. **Rademacher Complexity Terms**: The Rademacher Complexity terms in the theorem quantify the complexity of the hypothesis space. These complexity terms denote an intrinsic error bound that persists regardless of sample size, reflecting the capacity of the function class we are choosing from. This intrinsic error is a fundamental aspect of the learning problem and remains even in a fully supervised scenario [1].

Together, these components suggest that as we gather more data, the classifier $ \hat f_{sp} $ becomes increasingly accurate, drawing nearer to the performance of the classifier $ f^* $ within an acceptable error tolerance. The empirical risk for $ \hat{f}_{sp} $ is thus expected to converge to the true risk $ R(f^*) $.

   [1] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012. ISBN 978-0-262-01825-8.

Reviewer comment (Evaluation):
I am concerned about the time cost of the proposed method.

Author response:
Thank you for your comment regarding the computational efficiency of our proposed method. We understand the importance of time cost in real-world applications and have addressed this concern in two ways in our additional experiments.

   Firstly, we have recorded the time spent on estimating class priors within each epoch and found it to be a relatively minor portion of the total training time. As illustrated in the below table, the time for class-priors estimation is indeed quite short compared to the overall training time for an epoch, ensuring that our method remains practical for use in larger datasets.

   | Dataset | Time of Estimation of the Class-Priors (min) | Whole Training Time of One Epoch (min) |
   | :-----: | :------------------------------------------: | :------------------------------------: |
   |   VOC   |                     0.24                     |                  2.19                  |
   |  COCO   |                     3.47                     |                 27.29                  |
   |   NUS   |                     6.4                      |                 49.09                  |
   |   CUB   |                     0.45                     |                  3.89                  |

   Secondly, to further enhance the speed of our algorithm, we have experimented with updating the class priors every few epochs instead of every single one. The variant of our method, denoted as CRISP-3EP, updates the priors every three epochs and our experiments show that this results in a negligible loss in performance, as evidenced by the close metrics between CRISP and CRISP-3EP.

   |           |     VOC      |     COCO     |     NUS      |     CUB      |
   | :-------: | :----------: | :----------: | :----------: | :----------: |
   |   CRISP   | 89.820±0.191 | 74.640±0.219 | 49.996±0.316 | 21.650±0.178 |
   | CRISP-3EP | 89.077±0.251 | 73.930±0.399 | 49.463±0.216 | 19.450±0.389 |

   We will include this discussion in the limitations section in the revised manuscript. We believe these insights could be beneficial to researchers and practitioners using our method in the future.

Reviewer comment (Writing):
It is mentioned that "This unrealistic assumption will introduce severe biases into the pseudo-labels, further impacting the training of the model supervised by the inaccurate pseudo-labels". What are the biases? It is necessary to provide more discussions to enrich your motivations.

Author response:
The biases referred to in our paper arise from the discrepancy between the assumed uniform distribution of class-priors and the actual distribution in real-world data. Typically, the class distribution in real-world scenarios is imbalanced, with some classes being more prevalent than others. When pseudo-labels are generated under the assumption of equal class-priors, classes with a naturally lower occurrence rate are overrepresented, while those with a higher occurrence rate are underrepresented.

   The biases resulting from the unrealistic assumption of identical class-priors for pseudo-label generation can indeed lead to the production of incorrect pseudo-labels. This issue is pivotal, as the subsequent model training is supervised by these inaccurate pseudo-labels, which can compound the initial bias into a significant performance degradation.

Reviewer comment (Reproducibility):
The key of the proposed methods is the threshold. How do you get the optimal threshold in practice, i.e. how do you implement eq.2?

Author response:
In practice, to determine the optimal threshold, we conduct an exhaustive search across the set of outputs generated by the function $ f^j $ for each class. For instance, for a given class $ j $, and a set of instances $ \boldsymbol{x}_1, \boldsymbol{x}_2, \boldsymbol{x}_3 $ in our dataset, we compute the corresponding outputs $ z_1 = f^j(\boldsymbol{x}_1), z_2 = f^j(\boldsymbol{x}_2), z_3 = f^j(\boldsymbol{x}_3) $.

   The optimal threshold $ \hat{z} $ is then selected by identifying the value of $ z\in\\{z_1, z_2, z_3\\} $ that minimizes the objective function specified in Equation (2):

   $$
   \hat{z} = \arg\min_{z\in \\{z_1, z_2, z_3\\}} \left( \frac{\hat{q}_j(z)}{\hat{q}_j^p(z)} + \frac{1+\tau}{\hat{q}_j^p(z)}\left( \sqrt{\frac{\log(4/\delta)}{2n}} + \sqrt{\frac{\log(4/\delta)}{2n_j^p}} \right) \right)
   $$
   This approach ensures that we find the optimal threshold that minimizes the given expression, as per Equation (2), across all available output values from the function $f^j$.

Reviewer comment (Experiments):
The paper is motivated by the observation that previous methods have a strong assumption that class priors are assumed to be uniform. It would be interesting to see if the proposed method is still advantageous when class priors are uniform. It would enhance the paper's significance if the authors could demonstrate whether their proposed method retains its advantages even under the condition of uniform class priors.

Author response:
Thank you for your insightful query regarding the performance of our proposed method in scenarios where class priors are uniform. We agree that evaluating our method under this condition would provide a more comprehensive understanding of its robustness and versatility.

   However, unlike multi-class single-label datasets where the number of instances per class can be artificially balanced, multi-label datasets often exhibit complex label correlations that make such balancing more challenging. Existing multi-label datasets typically do not have uniform class priors, and due to the label correlations, balancing the occurrence of one class may inadvertently affect the distribution of other classes, making it impractical to achieve uniformity across all classes simultaneously [1].

   This inherent imbalance in multi-label settings underscores the importance of considering class-prior imbalance in the design of SPMLL methods. Our approach is particularly suited to address these real-world scenarios where uniform class priors are not the norm.

   [1] Wu, T., Huang, Q., Liu, Z., Wang, Y., & Lin, D. (2020). Distribution-balanced loss for multi-label classification in long-tailed datasets. In Proceedings of the 16th European Conference on Computer Vision (ECCV 2020) (pp. 162-178). Springer International Publishing.

Reviewer comment (Novelty):
It seems to me that the problem setting of SPMLL is a special case of "Multi-Label Ranking From Positive and Unlabeled Data" (CVPR 2016). I wonder if these general methods can be used as a baseline (and if not, what are the weaknesses of using these more general methods?).

Author response:
I agree that the problem setting of SPMLL is a special case of "Multi-Label Ranking From Positive and Unlabeled Data" (It is also called Multi-label learning with missing labels, MLML). We acknowledge the reviewer's suggestion to compare against MLML methods. As recommended, we have added experiments with the state-of-the-art MLML approach [1] on SPMLL datasets. The results show our proposed method still achieves superior performance, as shown in below results:

   Predictive performance of each comparing method on four MLIC datasets in terms of *mAP* (mean ± std). The best performance is highlighted in bold (the larger the better).

   |       |       VOC        |       COCO       |     NUSWIDE      |       CUB        |
   | :---: | :--------------: | :--------------: | :--------------: | :--------------: |
   | LL-R  |   87.784±0.063   |   70.078±0.008   |   48.048±0.074   |   18.966±0.022   |
   | LL-Cp |   87.466±0.031   |   70.460±0.032   |   48.000±0.077   |   19.310±0.164   |
   | LL-Ct |   87.054±0.214   |   70.384±0.058   |   47.930±0.010   |   19.012±0.097   |
   | Crisp | **89.820±0.191** | **74.640±0.219** | **49.996±0.316** | **21.650±0.178** |

   Predictive performance of each comparing method on MLL datasets in terms of *Average Precision* (mean ± std). The best performance is highlighted in bold (the larger the better).

   |       |      Image      |      Scene      |      Yeast      |     Corel5k     |    Mirflickr    |    Delicious    |
   | :---: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
   | LL-R  |   0.605±0.058   |   0.714±0.035   |   0.658±0.006   |   0.268±0.002   |   0.625±0.001   |   0.296±0.004   |
   | LL-Cp |   0.595±0.031   |   0.735±0.028   |   0.700±0.000   |   0.259±0.004   |   0.621±0.007   |   0.251±0.007   |
   | LL-Ct |   0.600±0.012   |   0.669±0.052   |   0.629±0.007   |   0.258±0.004   |   0.619±0.004   |   0.253±0.004   |
   | Crisp | **0.749±0.037** | **0.795±0.031** | **0.758±0.002** | **0.304±0.003** | **0.628±0.003** | **0.319±0.001** |

Predictive performance of each comparing method on MLL datasets in terms of *Ranking Loss* (mean ± std). The best performance is highlighted in bold (the smaller the better).

   |       |      Image      |      Scene      |      Yeast      |     Corel5k     |    Mirflickr    | Delicious       |
   | :---: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | --------------- |
   | LL-R  |   0.346±0.072   |   0.155±0.021   |   0.227±0.001   |   0.114±0.001   |   0.123±0.003   | 0.129±0.002     |
   | LL-Cp |   0.329±0.041   |   0.148±0.017   |   0.215±0.000   |   0.114±0.003   |   0.124±0.003   | 0.160±0.001     |
   | LL-Ct |   0.327±0.019   |   0.180±0.038   |   0.238±0.001   |   0.115±0.001   |   0.124±0.002   | 0.160±0.000     |
   | Crisp | **0.164±0.027** | **0.112±0.021** | **0.164±0.001** | **0.113±0.001** | **0.118±0.001** | **0.122±0.000** |

   Predictive performance of each comparing method on MLL datasets in terms of *One Error* (mean ± std). The best performance is highlighted in bold (the smaller the better).

   |       |      Image      |      Scene      |      Yeast      |     Corel5k     |    Mirflickr    |    Delicious    |
   | :---: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
   | LL-R  |   0.597±0.084   |   0.490±0.054   |   0.436±0.087   |   0.715±0.006   |   0.342±0.016   |   0.543±0.041   |
   | LL-Cp |   0.629±0.043   |   0.450±0.051   |   0.240±0.000   |   0.731±0.016   |   0.357±0.016   |   0.490±0.028   |
   | LL-Ct |   0.616±0.019   |   0.574±0.074   |   0.552±0.097   |   0.726±0.022   |   0.375±0.012   |   0.475±0.019   |
   | Crisp | **0.325±0.026** | **0.311±0.047** | **0.227±0.004** | **0.646±0.006** | **0.295±0.009** | **0.402±0.003** |

   In conclusion, while MLML methods could be applied to SPMLL, our experiments validate that our approach designed specifically for the SPMLL problem outperforms these more general techniques.

   [1] Kim, Y., Kim, J. M., Akata, Z., & Lee, J. (2022). Large loss matters in weakly supervised multi-label classification. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 14156-14165).

Reviewer comment (Theory):
It would be helpful to explicitly write out the definition of the absolute loss function and the derivation in Eq. 5.

Author response:
Thank you for your valuable suggestion to explicitly define the absolute loss function and provide the derivation for Eq. 5. We acknowledge that these additions will enhance the clarity and comprehensiveness of our manuscript.

   The absolute loss function, which is used in our framework, is defined as $l(f^j(x),y_j)=|f^j(x)-y_j|$. And Eq. 5 is derived as:
$$
\begin{aligned}
\mathcal{R}(f) &= \sum_{\boldsymbol y}p(\boldsymbol y) \mathbb{E}_{\boldsymbol x \sim p(\boldsymbol x \vert \boldsymbol y)} \left[ \sum _{j=1}^c  y_j\ell(f^j(\boldsymbol x), 1) + (1 - y_j)\ell(f^j(\boldsymbol x), 0) \right] \\\\
& = \sum _{j=1}^c p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[ \ell(f^j(\boldsymbol x), 1) \right] + p(y_j = 0) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 0)}\left[ \ell(f^j(\boldsymbol x), 0) \right] \\\\
& = \sum _{j=1}^c p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[ 1 - f^j(\boldsymbol x) \right] + (1 - p(y_j = 1)) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 0)}\left[ f^j(\boldsymbol x) \right] \\\\
& = \sum _{j=1}^c p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[ 1 - f^j(\boldsymbol x) \right] + \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x)}\left[f^j(\boldsymbol x)\right] - p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[f^j(\boldsymbol x) \right] \\\\
& = \sum _{j=1}^c p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[ 1 - f^j(\boldsymbol x) \right] + \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x)}\left[f^j(\boldsymbol x)\right] - p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[f^j(\boldsymbol x) - 1 + 1 \right] \\\\
& = \sum _{j=1}^c 2p(y_j = 1) \mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x \vert y_j = 1)}\left[ 1 - f^j(\boldsymbol x) \right] + \left(\mathbb{E} _{\boldsymbol x \sim p(\boldsymbol x)}\left[f^j(\boldsymbol x)\right] - p(y_j = 1)\right).
\end{aligned}
$$
   We ensure that these elements will be meticulously detailed in the revised version of the manuscript.

Reviewer comment (Theory):
I wasn't sure if we end up with an unbiased estimator (even with access to the ground truth class prior), because we have the additional absolute operator in the latter half of Eq. 7. It would be helpful if the paper can clarify.

Author response:
Thank you for your query regarding the unbiasedness of our estimator, particularly concerning the incorporation of the absolute value operator in Eq. 7. In our revision, we will clarify that the property of risk consistency with the absolute loss function is contingent upon the condition that $ E_{x\sim p(x)}[f^j(x)] \geq p(y_j=1) $. This condition ensures that the expected output of our model for class $ j $ is not less than the ground truth prior probability of that class being the positive label. 

   The inclusion of the absolute value serves two purposes in our formulation. On one hand, it ensures that when $ E_{x\sim p(x)}[f^j(x)] < p(y_j=1) $, the expected value $ E_{x\sim p(x)}[f^j(x)] $ is coerced to increase, thereby preserving the desirable property of risk consistency for the risk estimator. On the other hand, it aligns the model's output class prior with the true class prior, which is in direct correlation with the initial motivation of our paper—addressing the assumption of uniform class priors in previous methodologies.

Reviewer comment (Theory):
The algorithm relies on iteratively estimating the class-prior from f and then using it to update f. Is it possible if there is a failure mode?

Author response:
Our approach initially treats all unknown labels as negative to warm up the model, which is a common practice in existing SPMLL methods. This warming-up step provides a stable starting point, yielding a reasonably effective model before the application of our proposed Crisp method. Upon employing the CRISP framework, we iteratively refine the model through the class-prior estimation technique. In our extensive experimental evaluations, we did not encounter instances of failure.

Reviewer comment (Theory):
Would it be possible to extend this type of estimator to a different loss than the absolute loss?

Author response:
In our proposed framework, the risk estimator's extension to loss functions beyond absolute loss is indeed feasible, specifically for symmetric loss functions. A symmetric loss function $l$ has the property that for any prediction $f^j(x)$, the sum of the loss for a positive label and the loss for a negative label is constant, i.e., $l(f^j(x),+1)+l(f^j(x),0)=C$, where $C$ is a constant.