Paper: Learning with Complementary Labels Revisited: A Consistent Approach via Negative-Unlabeled Learning (ICLR-2024)

Reviewer comment (Writing):
The notations are confused, e.g., $\overline{Y}$ is a set of label vectors composed of {0,1}, and $k$ denotes the number of classes in {1,2, ..., q}, however, on page 4, line 5, the authors claim that "$k\in\overline{Y} $". The notations should be carefully checked and standardized.

Author response:
**A1:** Sorry for the confusion, but the notation in the paper is correct. The paper uses two notations to denote the complementary label set for each example, $\bar{Y}$ and $\bar{\boldsymbol{y}}$. Here, $\bar{Y} \subseteq \mathcal{Y}$ is a **set of labels** which is a subset of all the labels in the label space $\mathcal{Y}$, not a set of label vectors. Besides, $\bar{\boldsymbol{y}}=\left[\bar{y}\_{1},\bar{y}\_{2},\ldots,\bar{y}\_{q}\right] \in \\{0, 1\\}^q$ is a single label vector that also denotes the complementary labels of this example, where $\bar{y}\_{k}=1$ if $k \in \bar{Y}$ and $\bar{y}\_{k}=0$ otherwise. We refer to the notation used in multi-label classification, where each example is associated with multiple true labels [1].

Reviewer comment (Evaluation):
Although the experimental results of the proposed method in the paper are better than the compared methods, there is a lack of experimental analysis provided. A thorough analysis of the experimental results would be helpful in understanding the factors contributing to the superior performance.

Author response:
**A2:** Yes, adding experimental analysis is very helpful. We have added a detailed analysis of the experimental results in Appendix J in the updated version of our paper, and we list them here.

On synthetic benchmark datasets, we can observe
- Out of 40 cases of different distributions and datasets, CONU achieves the best performance in 39 cases, which clearly validates the effectiveness of the proposed approach.
- Some consistent approaches based on the uniform distribution assumption can achieve comparable or better or comparable performance to CONU for the "uniform" setting. For example, GA outperforms CONU on CIFAR-10. However, its performance drops significantly on other distribution settings. This shows that methods based on the uniform distribution assumption cannot handle more realistic data distributions.
- The variance of the classification accuracies of CONU is much smaller than that of the compared methods, indicating that CONU is very stable.
- The strong performance of CONU is due to the milder distribution assumptions and the effectiveness of the risk correction function in mitigating overfitting problems.

On real-world benchmark datasets, we can observe that
- CONU achieves the best performance in all cases, further confirming its effectiveness.
- The superiority is even more evident on CLCIFAR-20, a more complex dataset with extremely limited supervision. It demonstrates the advantages of CONU in dealing with real-world datasets.
- The performance of many state-of-the-art partial-label learning methods degenerates strongly, and many methods did not even work on CLCIFAR-20. The experimental results reveal their shortcomings in handling real-world data.

Reviewer comment (Experiments):
Although this paper provides nice theoretical results, it does not explain why the proposed method performs well. Which part plays an important role, the one-versus-rest (OVR) strategy, the risk correction function, or any other techniques?

Author response:
**A3:** It is exciting to observe the strong performance of our method because we did not use any strong regularization techniques or even tune the hyperparameters. We believe that the effectiveness is due to: 1) CONU is based on a milder data distribution assumption. We do not rely on the uniform data distribution or an ordinary-label dataset. Therefore, our method can handle more complex data distributions in more realistic scenarios. 2) The risk correction function is very effective in mitigating overfitting problems. From Figure 1, we can see that the loss function without the risk correction function encounters severe overfitting problems. However, with the help of the risk correction function, the performance is greatly improved.

Reviewer comment (Theory):
The details of the constant $c_k$ in assumption 1 are missed. How to decide this constant? Will the choice of this constant affect the performance?

Author response:
**A4:** Thanks for your question. We have added in Assumption 1 that $c_k=\bar{\pi}\_{k}/(1-\pi_k)$. Therefore, $c_k$ represents the fraction of complementary-label data with the $k$-th class as a complementary label in all the examples not belonging to the $k$-th class. The value of $c_k$ is determined by how many complementary labels are annotated. If more complementary labels are given by the annotator, $c_k$ will be larger. Then, the performance of complementary-label learning methods will be better since more supervision information is given.

Reviewer comment (Theory):
Why set the label $\bar{y}$ as a q-dimensional vector? Intuitively, adding more complementary labels would improve the performance.

Author response:
**A5:** We use this symbol only for ease of notation, since we need the marginal densities where the $k$-th class is considered as complementary or not in our formulation, i.e. $p\left(\boldsymbol{x}|\bar{y}\_{k}=1\right)$ and $p\left(\boldsymbol{x}|\bar{y}\_k=0\right)$. We believe this notation is clearer. Also, of course, adding more complementary labels can improve performance. If we add more complementary labels, more elements in $\bar{\boldsymbol{y}}$ will change from 0 to 1.

Reviewer comment (Theory):
The assumptions in Theorem 2 directly borrow from Theorem 10 of [1]. According to these assumptions, which functions can be used as the loss function $l$? Can you list some of these and write the explicit form of $l$ that can be used in practice?

Author response:
**A6:** Any binary-class loss function $\ell$ that is convex, bounded below, differential, and satisfies $\ell(z)\leq \ell(-z)$ when $z>0$ can make Theorem 2 hold. We list a few most common loss functions below:
|Name|$\ell(z)$|
|---|---|
|Squared|$(1-z)^2$|
|Logistic|$\log(1+\exp(-z))$|
|Exponential|$\exp(-z)$|

Reviewer comment (Evaluation):
The compared method GA[2] also adopts a risk correction approach, can you explain why your CONU outperforms GA?

Author response:
**A7:** We can see that on CIFAR-10 with the uniform distribution, GA outperforms CONU. This shows that we cannot rely on one approach for all datasets, and each approach has its own advantage. However, on other distribution settings, CONU outperforms GA because GA relies on the uniform distribution assumption while CONU relies on a milder assumption. The performance of GA degenerates significantly when the complementary labels do not follow the uniform distribution.

Reviewer comment (Experiments):
The methodologies compared in this research are outdated, if not obsolete. I'm afraid that most cutting-edge techniques are missing.

Author response:
**C3: The methodologies compared in this research are outdated. I'm afraid that most cutting-edge techniques are missing.**

**A3:** We included the most recent CLL and PLL approaches in 2023, i.e., OP [3] and IDGP [1], respectively, so we are very confused about what the "most cutting-edge techniques" are.

Reviewer comment (Writing):
The loss proposed for complementary label learning is uncear.

Author response:
On page 6, line 4, we said
> We refer to the method that works by minimizing the corrected risk estimator in Eq. (9) introduced below as CONU, where the algorithm details are summarized in Appendix B.

In the first two lines in Appendix H.3, we said
> The logistic loss was adopted to instantiate the binary-class loss function $\ell$, and the absolute value function was used as the risk correction function $g$ for CONU.

Therefore, the proposed loss is **very clear** in this paper.

Reviewer comment (Experiments):
The experiments are conducted on simple datasets, some complex dataset like cifar100, subset of webvision should be used to verify the effectness of the proposed method.

Author response:
**A5:** Contrary to your comment, we did include experiments on a variant of CIFAR-100 in the paper. We used CLCIFAR-20, a recent CLL benchmark based on CIFAR-100 with superclass labels. Therefore, **this comment is also unreasonable**.

**A6:** We still respectfully **disagree** with this comment. We didn't use WebVision because it is a noisy-label learning dataset, we cannot ensure that the complementary label generated is not the true label. Thus, no previous work in PLL or CLL takes this into account, and we simply followed their protocol. We would appreciate it if you could **provide a reference** in the CLL literature that uses this dataset to support your comment.

Reviewer comment (Novelty):
Is this strategy of this paper is similar to a previous work of Ishida2017[1].

[1] Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In Advances in Neural Information Processing Systems 30, pp. 5644–5654, 2017.

Author response:
Please allow me to list these two loss functions.

The loss function of OVA [4]:
\begin{equation}
\widehat{R}(f)=\frac{K-1}{n} \sum_{i=1}^n \overline{\mathcal{L}}(f(\boldsymbol{x}\_i), \bar{y}\_i)-M_1+M_2, 
\end{equation}
where 
\begin{equation}
\overline{\mathcal{L}}(f(\boldsymbol{x}), \bar{y})=\frac{1}{K-1} \sum_{y \neq \bar{y}} \ell\left(g_y(\boldsymbol{x})\right)+\ell\left(-g_{\bar{y}}(\boldsymbol{x})\right).
\end{equation}
The loss function of CONU:
\begin{equation}
\widetilde{R}\left(f_{1}, f_{2}, \ldots, f_{q}\right)=\sum_{k=1}^{q}\widetilde{R}\_{k}(f_{k}),
\end{equation}
where 
\begin{equation} 
\widetilde{R}\_{k}(f_{k})=g\left(\widehat{R}^{\rm P}\_{k}(f\_{k})\right)+\frac{1-\pi_{k}}{n^{\rm N}\_{k}}\sum_{i=1}^{n^{\rm N}\_{k}}\ell\left(-f_{k}(\boldsymbol{x}\_{k,i}^{\mathrm{N}})\right)
\end{equation}
and
\begin{equation}
\widehat{R}^{\rm P}\_{k}(f_{k})=\frac{\bar{\pi}\_{k}+\pi_{k}-1}{n^{\rm N}\_{k}}\sum_{i=1}^{n^{\rm N}\_{k}}\ell\left(f_{k}(\boldsymbol{x}\_{k,i}^{\mathrm{N}})\right)+\frac{1-\bar{\pi}\_{k}}{n^{\rm U}\_{k}}\sum\_{i=1}^{n^{\rm U}\_{k}}\ell\left(f_{k}(\boldsymbol{x}\_{k,i}^{\mathrm{U}})\right).
\end{equation}
It is very clear that the two losses are completely different, even though they can both use the logistic loss to instantiate $\ell$. Apart from the loss function, our differences are: 1) OVA is only designed for learning with single complementary labels, while CONU can handle any number of complementary labels; 2) The data distribution assumptions are different. OVA is based on the uniform distribution assumption, while CONU is based on the SCAR assumption. Therefore, our method is not similar to OVA at all.

Reviewer comment (Experiments):
The performance of prior estimation should be evaluated in the empirical study.

Author response:
**A2:** Thank you for your suggestion. We have presented the experimental results of the proposed class prior estimation approach in Appendix A. We assumed that the ground-truth class priors for all labels and datasets are 0.1, which means that the test set was balanced. We generated complementary labels using the SCAR assumption with $\bar{\pi}\_{k}=0.5$. Here are the experimental results:

|Label Index| 1| 2| 3| 4| 5| 6| 7| 8| 9| 10|
|---|---|---|---|---|---|---|---|---|---|---|
|MNIST|0.104±0.011|0.119±0.012|0.110±0.009|0.099±0.008|0.101±0.010|0.087±0.007|0.089±0.005|0.106±0.019|0.091±0.008|0.096±0.016|
|Kuzushiji-MNIST|0.108±0.026|0.098±0.011|0.087±0.012|0.104±0.004|0.101±0.021|0.095±0.010|0.105±0.025|0.095±0.007|0.094±0.016|0.113±0.035|
|Fashion-MNIST|0.091±0.016|0.118±0.005|0.090±0.024|0.090±0.009|0.077±0.020|0.117±0.007|0.070±0.010|0.114±0.023|0.117±0.016|0.117±0.016|
|CIFAR-10|0.085±0.016|0.102±0.039|0.073±0.019|0.109±0.047|0.100±0.031|0.098±0.013|0.115±0.023|0.120±0.033|0.097±0.041|0.100±0.013|


We can observe that the class priors are accurately estimated in general with the proposed method.

Reviewer comment (Evaluation):
I notice some results of existing methods are much worse than the results reported in their original papers. For example, the results of NN and GA on K-MINST and F-MNIST in paper [1] are much higher than that are reported in your paper.

Author response:
**A3:** The different experimental results are due to different experimental settings. The main difference is in the hyperparameters. We believe that it is not trivial to tune hyperparameters in complementary-label learning without an ordinary-label dataset. A common practice in machine learning is to separate the validation set from the training set. However, since the ground-truth labels of complementary-label data are unknown, it is difficult to use a complementary-label validation set to tune hyperparameters. Therefore, for a fair comparison, we did not tune specific hyperparameters and fixed the hyperparameters for all compared methods to the same values. For example, the learning rate was set to 1e-3. Details of the hyperparameters can be found in Appendix H. In paper [2], the authors said "we used the Adam optimizer with a learning rate selected from {1e-1, 1e-2, 1e-3, 1e-4, 1e-5} and trained the models for 300 epochs". So we assumed that they used an ordinary-label dataset to tune the hyperparameters, which is why their results were better.

Reviewer comment (Experiments):
Have you tried FORWARD [2] on CLCIFAR datasets? I notice that the results of FORWAD is pretty good on these instance-dependent CL datasets and should be involved in the comparison.

Author response:
**A4:** As you suggested, we have added the experimental results of Forward on instance-dependent CL datasets. In A1, we have added experimental results of Forward on synthetic instance-dependent CL datasets. The comparison with CONU on CLCIFAR-10 and CLCIFAR-20 is done here. Since there are three complementary labels for each example, following [3], we used them separately to generate single complementary-label datasets. Since the transition matrix for Forward is unknown, we assume it to be a uniform distribution for a fair comparison. Here are the experimental results:

|Dataset|CL Index|Model|Forward|CONU|
|---|---|---|---|---|
|CLCIFAR-10|1|ResNet|16.71|**33.42**|
|CLCIFAR-10|1|DenseNet|17.70|**37.53**|
|CLCIFAR-10|2|ResNet|16.37|**32.27**| 
|CLCIFAR-10|2|DenseNet|16.67|**36.62**| 
|CLCIFAR-10|3|ResNet|16.58|**34.84**|
|CLCIFAR-10|3|DenseNet|17.15|**36.41**| 
|CLCIFAR-20|1|ResNet|5.00|**13.34**| 
|CLCIFAR-20|1|DenseNet|5.00|**13.94**| 
|CLCIFAR-20|2|ResNet|5.00|**13.22**| 
|CLCIFAR-20|2|DenseNet|5.00|**13.24**| 
|CLCIFAR-20|3|ResNet|5.00|**12.04**| 
|CLCIFAR-20|3|DenseNet|5.00|**10.91**|   

We can observe that CONU performs better than Forward on the CLCIFAR datasets.

Reviewer comment (Novelty):
The major novelty lies in the problem reformulation. The way to conduct theoretical analysis and risk correction is off-the-shelf. In this sense, the technical contribution is not very impressive.

Author response:
**A1:** We would like to clarify that the technical contribution and the focus of our paper are not only the theoretical results. The theoretical results are just auxiliaries to show the solid theoretical guarantees of our proposed methods, but not the main contributions. We have revised the contribution part of Introduction in the updated version of our paper to make it clearer.

First, we think that our attempt to get rid of the uniform distribution assumption and ordinary-label training sets is a **substantial contribution** to the CLL community, as also commented by Reviewer ye4F. All consistent CLL approaches (listed in Table 1) rely on the uniform distribution assumption or an ordinary-label training set, which are not realistic despite their solid theoretical properties. Therefore, designing a more realistic CLL approach with theoretical guarantees can make a great contribution and bring new insights to the field. 

Second, another major contribution is that we provide **a new way to understand CLL**. We uncover the relationship between CLL and negative-unlabeled learning, which is quite novel and critical to the CLL literature. To our knowledge, this relationship has never been explored in the CLL literature. We believe that this finding may provide new directions and insights for solving CLL problems. For example, in Remark 1, we explain that besides minimizing $R_{k}(f_k)$, we can use **any other PU learning approach** to derive the binary classifier $f_{k}$ by swapping the positive class and the negative class. Therefore, we have not only proposed a single loss function, but a **general framework that incorporates PU learning approaches to solve CLL problems**. Such a framework can also bring new insights to this field.

Therefore, we believe that our work is a significant contribution to the CLL community.

Reviewer comment (Experiments):
In Fig2, only the impact of inaccurate class priors over the proposed method is illustrated. How about the competitors?

Author response:
**A2:** CONU is the only CLL approach that uses class priors in the loss function. Therefore, the ground-truth class priors have no effect on the compared approaches.

Reviewer comment (Novelty):
Some recent PU-learning methods should be reviewed in Sec.2.2, e.g.,  
[1] Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends. NeurIPS 2023.  
[2] Positive-Unlabeled Learning With Label Distribution Alignment. TPAMI 2023.  
[3] GradPU: Positive-Unlabeled Learning via Gradient Penalty and Positive Upweighting. AAAI 2023.  
[4] Dist-PU: Positive-Unlabeled Learning From a Label Distribution Perspective. CVPR 2022.

Author response:
**A3:** Thanks for your suggestion. We have added these references and provided a review of PU learning in Appendix I in the updated version of our paper.

Reviewer comment (Writing):
A brief introduction for the compared methods to explain their respective characteristics will help understanding.

Author response:
**A4:** We agree with your comments and we have included the introductions in Appendix H in the updated version of our paper.