R OBUST
L OSS
F UNCTIONS
FOR
C OMPLEMENTARY
L ABELS L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In ordinary-label learning, the correct label is given to each training sample.
1 Similarly, a complementary label is also provided for each training sample in 2 complementary-label learning.
A complementary label indicates a class that the 3 example does not belong to.
Robust learning of classiﬁers has been investi4 gated from many viewpoints under label noise, but little attention has been paid 5 to complementary-label learning.
In this paper, we present a new algorithm of 6 complementary-label learning with the robustness of loss function.
We also pro7 vide two sufﬁcient conditions on a loss function so that the minimizer of the risk 8 for complementary labels is theoretically guaranteed to be consistent with the min9 imizer of the risk for ordinary labels.
Finally, the empirical results validate our 10 method’s superiority to current state-of-the-art techniques.
Especially in cifar10, 11 our algorithm achieves a much higher test accuracy than the gradient ascent algo12 rithm, and the parameters of our model are less than half of the ResNet-34 they 13 used.
I NSTRUCTION
Deep neural networks have exhibited excellent performance in many real-applications.
Yet, their 16 supper performance is based on the correctly labeled large-scale training set.
However, labeling 17 such a large-scale dataset is time-consuming and expensive.
For example, the crowd-workers need 18 to select the correct label for a sample from 100 labels for CIFAR100.
To migrate this problem, 19 reachers have proposed many solutions to learn from weak-supervision: Noise-label learning Li 20 et al.
; Hu et al.
; Lee et al.
; Xia et al.
(2019), semi-supervised learning Zhai 21 et al.
; Berthelot et al.
; Rasmus et al.
; Miyato et al.
; Sakai et al.
(2017), 22 similar-unlabeled learning Tanha (2019); Bao et al. ; Zelikovitz & Hirsh (2000), unlabeled23 unlabeled learning Lu et al. ; Chen et al. (2020a;b), positive-unlabeled learning Elkan & Noto 24 (2008);
du Plessis et al.
(2014); Kiryo et al.
(2017), contrast learning Chen et al. (2020a;b), partial 25 label learning Cour et al. Feng & An (2018); Wu & Zhang (2018) and others. We investigate complementary-label learning Ishida et al.
(2017) in this paper.
A complementary 27 Label is only indicating that the class label of a sample is incorrect.
In the view of label noise, 28 complementary labels can also be viewed as noise labels but without any true labels in the training 29 set.
Our task is to learn a classiﬁer from the given complementary labels, predicting a correct label 30 for a given sample.
Collecting complementary labels is much easier and efﬁcient than choosing a 31 true class from many candidate classes precisely.
For example, the label-system uniformly chooses a 32 label for a sample.
It has a probability of 1 k to be ordinary-label but k − 1 k to be complementary-label.
33 Moreover, another potential application of complementary-label is data privacy.
For example, on 34 some privacy issues, it is much easier to collect complementary-label than ordinary-label.
Robust learning of classiﬁers has been investigated from many viewpoints in the presence of label 36 noise Ghosh et al.
, but little attention paid to complementary-label learning.
We call a loss 37 function robust if the minimizer of risk under that loss function with complementary labels would be 38 the same as that with ordinary labels.
The robustness of risk minimization relies on the loss function 39 used in the training set.
This paper presents a general risk formulation that category cross-entropy loss (CCE) can be used to 41 learn with complementary labels and achieve robustness.
We then offer some innovative analytical 42 results on robust loss functions under complementary labels.
Having robustness of risk minimization  helps select the best hyper-parameter by empirical risk since there are no ordinary labels in thevalidation set.
We conclude two sufﬁcient conditions on a loss function to be robust for learningwith complementary labels.
We then explore some popular loss functions used for ordinary-labellearning, such as CCE, Mean square error (MSE) and Mean absolute error (MAE), and show that
CCE and MAE satisfy our sufﬁcient conditions.
Finally, we present a learning algorithm for learningwith complementary labels, named exclusion algorithm.
The empirical results well demonstrate theadvantage of the theoretical results we addressed and verify our algorithm’s superiority to the currentstate-of-the-art methods.
The contribution of this paper can be summarized as:
• We present a general risk formulation that can be view as a framework to employing a
loss function that satisﬁes our robustness sufﬁcient condition to learn from complementary
labels.
• We conclude two sufﬁcient conditions on a loss function to be robust for learning with
complementary labels.
• We prove that the minimizer of the risk for complementary labels is theoretically guaran-
teed to be consistent with the minimizer of the risk for ordinary labels.
• The empirical results validate the superiority of our method to current state-of-the-art meth-
ods.
R ELATED WORKS
Complementary-label refers to that the pattern does not belong to the given label.
Learning fromcomplementary labels is a new topic in supervised-learning.
It was ﬁrst proposed by Ishida et al.(2017).
They conduct such an idea to try to deal with time-consuming and expensive to tag a largescale dataset.
In their early work Ishida et al.
(2017), they assume the complementary labels are the same probability to be selected for a sample.
And then, based on the ordinary one-versus-all (OVA) andpairwise-comparison (PC) multi-class loss functions Zhang (2004) proposed a modifying loss forlearning with complementary labels.
Even though they provided theoretical analysis with a statistical consistency guarantee, the lossfunction met a sturdy restriction that needs to be symmetric ( (cid:96) ( z ) + (cid:96) ( − z ) = 1 ).
Such a severelimitation allows only the OVA and PC loss functions with symmetric non-convex binary losses.
However, the categorical cross-entropy loss widely used in the deep learning domain, can not beemployed by the two losses they deﬁned.
Later, Yu et al.
(2018a) assume there are some biased amongst the complementary labels andpresents a different formulation for biased complementary labels by using the forward loss correction technique Patrini et al.
(2017) to modify traditional loss functions.
Their suggested riskestimator is not necessarily unbiased and proved that learning with complementary labels can theoretically converge to the optimal classiﬁer learned from ordinary labels based on the estimatedtransition matrix.
However, the key to the forward loss correction technique is to evaluate the transition matrix correctly.
Hence, one will need to assess the transition matrix beforehand, which isrelatively tricky without strong assumptions.
Moreover, in such a setup, it restricts a small complementary label space to provide more information.
Thus, it is necessary to encourage the workerto provide more challenging complementary labels, for example, by giving higher rewards to thespeciﬁc classes.
Otherwise, the complementary label given by the worker may be too evident anduninformative.
For example, class three and class ﬁve are not class one evidently but is uninformative.
This paper focuses on the uniform (symmetric) assumption and study random distribution as abiased assumption (asymmetric or non-uniform).
Based on the uniform assumption, Ishida et al.
(2019) proposed an unbiased estimator with a generalloss function for complementary labels.
It can make any loss functions available for use, not onlysoft-max cross-entropy loss function, but other loss functions can also be utilized.
Their new framework is a generalization of previous complementary-label learning Ishida et al.
However,their proposed unbiased risk estimator has an issue that the classiﬁcation risk can attain negativevalues after learning, leading to overﬁtting Ishida et al.
They then offered a non-negativecorrection to the original unbiased risk estimator to improve their estimator, which is no longer guaranteed to be an unbiased risk estimator. In this paper, our proposed risk estimator is also not 96 unbiased, but the minimizer of the risk for complementary labels is theoretically guaranteed to be 97 consistent with the minimizer of the risk for ordinary labels, both uniform and non-uniform.
P RELIMINARIES
3. L EARNING WITH ORDINARY LABELS
In the context of learning with ordinary labels, let X ⊂ R d be the feature space and Y = { 1 , · · · , k }be the class labels.
A multi-class loss function is a map: L ( f θ ( x ) , y ) :
X × Y → R + .
A classiﬁercan be presented as:
[Equation]
where f θ ( x ) = ( f (1) θ ( x ) , · · · , f ( k ) θ ( x )) , θ is the set of parameters in the CNN network, f ( i ) θ ( x ) isthe probability prediction for the corresponding class i .
Even though h ( x ) is the ﬁnal classiﬁer, weuse notation of calling f θ ( x ) itself as the classiﬁer.
Given dataset S
= { ( x i , y i ) } Ni , together with aloss function L , ∀ f θ ∈ F ( F is the function space for searching), L -risk is deﬁned as:
[Equation]
Some popular multi-class loss functions are CCE, MAE, MSE.
Speciﬁcally,
[Equation]
[Equation]
where u = f θ ( x ) =
( µ 1 , · · · , µ k ) , and e y is a one-hot vector that the y -th component equals to 1,others are 0.
The goal of multi-class classiﬁcation is to learn a classiﬁer f θ ( x ) that minimize theclassiﬁcation risk R SL with multi-class loss L .
3. L EARNING WITH COMPLEMENTARY LABELS
In contrast to the ordinary-label learning, the complementary-label (CL) dataset contains only labelsindicating that the class label of a sample is incorrect.
Corresponding to the ordinary labels dataset S , the independent and identically distributed (i.i.d.) complementary labels dataset denoted as:
[Equation]
where N is the size of the dataset ¯ S , and ¯ y represents that pattern x does not belong to class- ¯ y .
The general labels’ distribution of dataset ¯ S is as:
[Equation]
where p ij denotes that the probability of the i -th class’s pattern x labeled as j , (cid:80) kj =1 p ij =1 , p ij (cid:54) =0 , j (cid:54)
=
i .
Supposing that the label system uniformly select a label from { 1 , · · · , k } \ { y }for each sample x , then the Eq.
(5) becomes
[Equation]

[Equation]
Yu et al.
(2018b) make a strong assumption that there are some bias in Eq.
(5), while Ishidaet al.
(2017; 2019) focus on the assumption of Eq.
In this paper, we study both kinds of distribution.
M ETHODOLOGY
In this section, we ﬁrstly propose a general risk formulation for leaning with complementary labels.
And then prove that some loss functions designed for the ordinary labels learning are robust tocomplementary labels with our risk formulation, such as categorical cross-entropy loss and meanabsolute error.
4. G ENERAL RISK FORMULATION
The goal of learning with complementary labels is to learn a classiﬁer that predicts a correct label forany sample drawn from the same distribution.
Because there are not ordinary labels for the model,we need to design a loss function or model for learning with complementary labels.
The key to learning a classiﬁer for ordinary label learning is to maximize the true label’s predict-probability.
Oneintuitive way to maximize the true label’s predict-probability is to minimize the predict-probabilityof complementary labels.
In this paper, with little abuse of notation, we let
[Equation]
Deﬁnition 1.
(CL-loss) Together with loss function (cid:96) designed for the ordinary-label learning, theloss for learning with complementary-label is as:
[Equation]
4. T HEORETICAL RESULTS
Deﬁnition 2.
(Robust loss)
In the framework of risk minimization, a loss function is called robustloss function if minimizer of risk with complementary labels would be the same as with ordinarylabels, i.e.,
[Equation]
Theorem 1.
Together with (cid:96) , ¯ (cid:96) is a robust loss function for learning with complementary labels, if ¯ (cid:96)satisﬁes:
[Equation]
Note that, in Eq.
10, it means that ¯ (cid:96) is a monotone increasing loss function only on u (¯ y ) .
Proof.
Recall that for any f θ , and any (cid:96) ,
[Equation] [Equation]
For any complementary-label distribution in Eq.
(5), and any loss function (cid:96) , we have
[Equation]
where p ij is the component of complementary labels distribution matrix P , S 1 ∪ · · · ∪ S k = S .
Supposing that f θ ∗ is the optimal classiﬁer learns from the complementary labels, and ∀ f ∈ F ,where F is the function space for searching, we have [Equation] p
[Equation]
(cid:0) ¯ (cid:96) ( f θ ∗ ( x ) [Equation] j ) − ¯ (cid:96) ( f θ ( x ) , j ) (cid:1) ≤ 0 , (13) where p ij (cid:54) = 0 .
If ∃ x (cid:48) ∈ ¯ S , s.t., ¯ (cid:96) ( f θ ∗ ( x (cid:48) ) , ¯ y ) > ¯ (cid:96) ( f θ ( x (cid:48) ) , ¯ y ) , let f θ (cid:48) satisfying [Equation]
[Equation]
then according to Eq.
12 and 13, R ¯ S ¯ (cid:96) ( f θ (cid:48) )
<
R ¯ S ¯ (cid:96) ( f θ ∗ ) , f θ ∗ is not the optimal classiﬁer.
Thiscontradicts the hypothesize that f θ ∗ is the optimal classiﬁer.
Thus, ∀ ¯ y ∈ { 1 , · · · , k } \ { y } , we have
[Equation]
According to Eq.
(10), ¯ (cid:96) is a monotone increasing loss function only on u (¯ y ) , then we have
[Equation] [Equation]
Thus,
[Equation]
and then,
[Equation]
[Equation]
thus,
[Equation]
Theorem 2.
Together with (cid:96) , ¯ (cid:96) is a robust loss function for learning with complementary labelsunder symmetric distribution or uniform distribution, if ¯ (cid:96) satisﬁes:
[Equation] [Equation]
is a constant) .
It should be noted that, in Eq. 20, it means that ¯ (cid:96) is a symmetric loss ( (cid:80) (cid:96) ( u , i ) = C ), and ¯ (cid:96) is amonotone increasing loss function on any ¯ y .
Proof.
For any complementary-label distribution in Eq.
(6), and any loss function (cid:96) , we have
[Equation]
[Equation]
where S 1 ∪ · · · ∪ S k = S .
Supposing that f θ ∗ is the optimal classiﬁer learns from the complementary labels, and ∀ f ∈ F ,where F is the function space for searching, we have
[Equation]
According to the ﬁrst constraint in Eq.
(20), we then have
[Equation]
(23) and then, 168 (cid:96)
[Equation]
24) thus,
[Equation]
[Equation]
Together with some well known multi-class loss functions, such as CCE, MAE, MSE, the loss forlearning with complementary labels with our deﬁnition are as follows:
[Equation]
where e ¯ y is a one-hot vector that the ¯ y -th component equals to 1, others are 0.
As its shown in
Eq.
(26), CCE and MAE loss satisfy the Theorem 1, MAE also satisﬁes the Theorem 2, while MSEdoes not satisﬁes the two.
Zhang & Sabuncu (2018) propose a GCE loss function for learning withlabel noise, their formulation is as:
[Equation]
It is easily to know that the loss function satisﬁes the constraint in Theorem 1, thus, it can be usedto learning with complementary labels.
4. E XCLUSION ALGORITHM FOR LEARNING FROM COMPLEMENTARY LABELS
Based on the loss function we designed for complementary-label learning, we present an algorithmto learn a classiﬁer from complementary labels with our loss function, named exclusion algorithm(the label speciﬁes that the sample does not belong to it).
The algorithm details show in Alg. 1.
Furthermore, our algorithm is easily combined with the models designed for ordinary-label learning,with only a minus operation, which can be view as a framework to use the loss designed for ordinarylabel learning to learn the optimal classiﬁer from complementary labels.
E XPERIMENTS
5. E XPERIMENTAL SETTINGS
Datasets.
We test our experiments on MNIST LeCun et al.
FASHION-MNIST Xiao et al.(2017), CIFAR10 Krizhevsky (2009).
Speciﬁcally, we generate two types of complementary labels:symmetric and asymmetric, for our experiments to verify our method’s effectiveness and the theoremwe proved in the previous section.
For symmetric complementary-label, we ﬁx a label distribution as
Eq.
(6) to generate the complementary-label training set.
The validation set is split from the trainingset, which contains none ordinary-label.
Thus, the lower the validation accuracy, the better theclassiﬁer learns from the training set.
For asymmetric complementary-label, we randomly generatea matrix as Eq.
(5) that the p ij is unknown as the complementary-label distribution and using it to create complementary-label for experiments.
The test accuracy of all experiments is tested on aclean dataset that contains only the ordinary labels.
Approaches.
We test our loss with ¯ (cid:96) CCE , ¯ (cid:96) MAE , ¯ (cid:96) MSE , ¯ (cid:96) GCE and compare with state-of-the-art methods in learning with complementary labels.
The loss functions we used or compare in this paperare listed as follows.
1) CCE: The categorical cross-entropy loss, neither symmetric nor bounded,which widely use in machine learning and deep learning due to its fast convergence speed.
2) MAE:
The mean absolute error, a symmetric loss and bounded, has been proved Ghosh et al.
(2017) to benoise-tolerant.
3) MSE:
The mean square error, not symmetric but bounded, widely used in regression learning.
GCE:
It uses a hyper-parameter q to tune the loss between MAE and CCE, butachieve noise-robust base on its bounded, we used the standard GCE where q=0.7 . 5) GA:
Gradient ascent, a learning algorithm for complementary-label learning, is used to tackle the overﬁttingproblem of the unbiased estimator they proposed in Ishida et al.
6) PC: Pairwise comparison (PC) with ramp loss designed for complementary-label learning Ishida et al.
7) Fwd:
Forward correction Patrini et al.
(2017), Yu et al.
(2018a) designed for learning with complementarylabels.
Network architecture.
Following Ghosh et al.
(2017), we use a network architecture that containsﬁve layers to test the above methods for all the experiments: a convolution layer with 32 ﬁlters whichﬁlter size set as (3,3), a max-pooling layer with pooling-size of (3,3) and strides of (2,2), two fullyconnected layers with 1024 units, and a fully connected layer with soft-max activated function thatthe unit number set to the category number for prediction.
Rectiﬁed Linear Unit (ReLU) is used asthe activated function in the network’s hidden layer.
Implement details.
The implementation detail of our method shows in Alg.
We train our networkwith stochastic gradient descent through back-propagation.
Each experiment trains 200 epochs, andthe mini-batch size was set to 64.
To exploit each loss function’s best performance, we set three startlearning rate for each loss function on each experiment and report the best accuracy amongst thethree learning rate of each loss function.
CCE is set to [1e-3, 5e-4, 1e-4], while GCE, MAE, MSEis set to [1.0, 0.5, 0.1].
The learning rate was halved per 50 epochs.
[Figure]
5. E XPERIMENTAL RESULTS
[Figure]
Robustness.
As shown in Fig.
1, together with CCE, MAE
[Figure]
and GCE loss, our algorithm achievesstrong robust to both symmetric and asymmetric complementary labels, which verify that the robustness we prove in the Theorem 1 and Theorem 2.
Even though the MAE satisﬁes the two theorems,
[Table]
it achieves a lower test accuracy than that of CCE and GCE due to it treats all labels the same (notsensitive to the labels).
The subﬁgures in the last column of Fig.
1 shows that the MSE loss ﬁrstlyachieves its highest test accuracy and then drop sharply over the epochs.
Because MSE does notsatisfy one of the two theorems we prove, it easily overﬁts the training set’s complementary labels.
Such a trend is the same as asymmetric complementary labels learning.
The results verify that thealgorithm we design for the complementary labels is signiﬁcant and conﬁrms the theoretical resultswe analyzed in the previous section.
Performance Comparison.
The ﬁrst four columns of Table.
1 show that the CCE and GCE lossachieve the best two test accuracies in our algorithm.
In the MNIST dataset, the CCE achievesa little lower test accuracy than GCE, the same test accuracy in FASHION-MNIST, and a littlehigher test accuracy in CIFAR10 due to the dataset more challenge and CCE is more sensitive tolabels.
Even MAE is robust to complementary labels, and its performance is not well than othersbecause it is a linear loss that is not sensitive to labels.
As shown in Fig.
1, MSE is not robust tocomplementary labels, but with a small learning rate of 0.1, MSE only exhibited slight overﬁtting in
Table 1.
Furthermore, as shown in Table 1, together with CCE and GCE loss, our algorithm achievesa test accuracy higher than 95% in the MNIST dataset, which is comparable to that of learning withordinary labels.
For a fair comparison, The last three columns directly form Ishida et al.
(2019) even those resultsare the max test accuracy.
In the ﬁrst two datasets, all loss functions with our algorithm achieve ahigher test accuracy than GA, but they used an MLP model as their base model, simpler than ours.
In CIFAR10, they used ResNet-34 (21.62M parameters)
He et al.
and
DenseNet Huang et al.(2017) as their based model, which is much bigger than ours (8.43M parameters), but we achieve amuch higher test accuracy than theirs.
The results validate the superiority of our algorithm to currentstate-of-the-art methods.
C ONCLUSION
This paper designs an algorithm for learning from complementary labels using the loss functions designed for ordinary-label learning.
We provide theoretical analysis to show that the loss func- tions we design for learning from the complementary labels are robust to the complementary labels, i e., the optimal classiﬁer learned from the complementary labels can theoretically converge to the optimal classiﬁer learned from ordinary labels.
In this paper, the two theorems we present are thesufﬁcient condition of a loss function robust to complementary labels.
Experimental results show that though complementary-label learning is a new topic in supervised-learning, it offers excellent competitiveness. More methods should be studied to improve the performance of complementary learning in our future works, such as Amid et al. (2019b) and
Amid et al. (2019a).