One Positive Label is Sufﬁcient: Single-Positive Multi-Label Learning with Label Enhancement
Abstract
Multi-label learning (MLL) learns from the examples each associated with multiple labels simultaneously, where the high cost of annotating all relevant labels for each training example is challenging for real-world applications.
To cope with the challenge, we investigate single-positive multi-label learning (SPMLL) where each example is annotated with only one relevant label and show that one can successfully learn a theoretically grounded multi-label classiﬁer on SPMLL training examples.
In this paper, a novel SPMLL method named S MILE , i.e., Singlepositive MultI-label learning with Label Enhancement, is proposed.
Speciﬁcally, an unbiased risk estimator is derived, which could be guaranteed to approximately converge to the optimal risk minimizer in fully supervised learning and shows that one positive label of each instance is sufﬁcient to train a model.
Then, the corresponding empirical risk estimator is established via recovering the latent soft label as a label enhancement process, where the posterior density of the latent soft labels is approximate to the variational Beta density parameterized by an inference model.
Experiments on twelve corrupted MLL datasets show the effectiveness of S MILE over several existing SPMLL approaches. Source code is available at https://github.com/palm-ml/smile .
Introduction
Multi-label learning (MLL) aims to build a predictive model to assign a set of relevant labels for the unseen instance via learning from the training examples associated with multiple class labels simultaneously [30, 46].
During the past decade, MLL has been widely applied to learn from the data containing rich semantics, such as multimedia content annotation [40, 33], text categorization [29, 27], music emotion analysis [21, 35], and bioinformatics analysis [3], etc.
However, in practice, obtaining ground-truth multiple labels for MLL training datasets is costly due to the expensive and time-consuming manual annotations.
Comparing with multi-class learning where an example is associated with only one positive label , multi-label learning requires the complete positive label set for each example.
On this account, the annotation cost of multi-label learning is signiﬁcantly higher than multi-class classiﬁcation, which limits its application especially when the number of categories is large.
To mitigate this problem, the setting of single-positive multi-label learning (SPMLL) [5] allows for signiﬁcantly reduced annotations costs for the datasets, where each example is annotated with [Figure]
only one relevant label. In Figure 1, comparing with fully labeled case, the SPMLL approaches on single-positive labeled examples only incur a tolerable drop in the performance but drastically reduce the amount of supervision required to train multi-label classiﬁers.
By establishing the SPMLL methods with the learning power of DNNs, recent work [5] has also empirically validated that SPMLL would reduce the annotations costs while achieving good performance in practice.
However, no method could provide theoretical insights as to why the model trained on the SPMLL examples can converge to an ideal one.
In this paper, we propose a theoretically-guaranteed method named S MILE , i.e., Single-positive
MultI-label learning with Label Enhancement.
Speciﬁcally, we ﬁrst derive an unbiased risk estimator, which suggests that one positive label of each instance is sufﬁcient to train the predictive models for multi-label learning.
Besides, an estimation error bound is derived, which guarantees the riskconsistency [25] of the proposed method.
Then we could design a benchmark solution via recovering the soft labels corresponding to each example in a label enhancement process [39, 36], where the posterior density of the latent soft labels is inferred by leveraging an approximate Beta density.
The contributions are summarized as follows:
• Theoretically, we for the ﬁrst time derive an unbiased risk estimator for SPMLL.
Based on this, an estimation error bound is established that guarantees the risk-consistency and demonstrates that the obtained risk minimizer in SPMLL would approximately converge to the optimal risk minimizer in fully supervised MLL.
Practically, we propose the method S MILE for SPMLL via adopting the latent soft labels recovered by label enhancement.
The posterior density of the latent soft label is inferred by leveraging an approximate Beta density and the evidence lower bound (ELBO) [18] for the optimization is deduced.
Experiments on twelve corrupted MLL datasets show the effectiveness of S MILE over several existing SPMLL approaches.
Related Work
In multi-label learning, each example is associated with multiple class labels simultaneously.
As the output space in MLL is exponential in size to the number of class labels, numerous approaches are proposed to exploit the label correlations to promote the learning process [14, 44, 31].
The ﬁrst-order approaches disassemble the MLL problem into a number of binary classiﬁcation problems [2, 45].
The second-order approaches consider the label correlations between pairs of labels [8, 11].
The high-order approaches further focus on the label correlations among the label set [26, 32].
Another line of research focuses on manipulating the feature space via formalizing label-speciﬁc feature to each class label to facilitate multi-label classiﬁcation [23, 15, 42].
In addition, some work focus on dealing with MLL via deep models.
A directed graph over the labels is established via employing GCN to propagate information among all label nodes [4].
Transformer is leveraged for exploring the label dependency by introducing a ternary encoding scheme to represent the state of labels [20].
In practice, the labeling information is often incomplete in training data, since acquiring exhaustive supervision is extremely difﬁcult.
Numerous approaches have been proposed to handle the MLL with missing labels [12], which is also termed as MLL with partial labels [7].
A transductive learning method is proposed to concatenate features and labels and apply the matrix completion technique to it [12].
Then, the inductive learning method is proposed to exploit the structure of speciﬁc loss functions to offer efﬁcient algorithms for learning with missing labels [41].
Wu [34] recovers the full labeling information of each training sample via enforcing consistency with available label-assignment and smoothness of label-assignment.
The global and local label correlations are exploited simultaneously via learning a latent label representation in the missing label cases [47].
Durand [7] empirically compare different label-assignment strategies to show the potential to employ partial labels for MLL.
Another method induces a cost function that measures the smoothness of labels and features to alleviate the overﬁtting issue when training data contains missing labels [16].
Comparing with multi-label learning with missing labels, SPMLL [5] considers the hardest version of this problem, where annotators are only asked to provide a single positive label for each training example and no additional negative or positive labels.
When collecting multi-label annotations, it may be more efﬁcient to annotate only one label rather than multiple labels for each example.
To learn from SPMLL examples, an intuitive solution is “assume negative” (AN) [5], which assumes that unobserved labels are negative and trains the predictive model with binary cross-entropy loss on observed positive labels.
Recent work [5] proposes some methods to reduce the damaging effects of false-negative labels.
An expected positive regularization [5] is proposed to avoid the problem but the expected number of positive labels of each example should be given.
Label smoothing [28] is employed to reduce the impact of the incorrect labels.
Another approach [5] estimates the unobserved labels and encourages the classiﬁer predictions to match the estimated labels via binary entropy loss.
However, no methods can provide theoretical insights as to why the model trained on SPMLL examples can converge to an ideal one.
Problem Setup 3.1 Multi-Label Learning
In MLL, each example is associated with multiple labels, and aims to build a predictive model which can assign a set of relevant labels for the unseen instance.
Let X
= R q be the q -dimensional instance space and Y = { 1 , 2 , ..., c } be the label space with c class labels.
Given the MLL training set D = { ( x i , Y i ) | 1 ≤ i ≤ n } where x i ∈
X denotes the q -dimensional instance and
Y i ∈ C is the set of relevant labels associated with x i where C = 2 Y .
The task of multi-label learning is to induce a multi-label classiﬁer f : X (cid:55)→ 2 Y that minimizes the following classiﬁcation risk:
[Equation]
Here, L : R q × 2 Y (cid:55)→ R + is a multi-label loss function that measures how well the model ﬁts the data.
Note that a method is risk-consistent if the method possesses a classiﬁcation risk estimator that is equivalent to R ( f ) given the same classiﬁer f
3.2 Single-Positive Multi-Label Learning
Given the SPMLL training set (cid:101) D
= { ( x i , γ i ) | 1 ≤ i ≤ n } where γ i ∈ Y denotes the observed single-positive label of x i .
Note that γ i ∈ Y i while its relevant label set Y i is not directly accessible to the learning algorithms.
For each SPMLL training example ( x i , γ i ) , we use the observed singlepositive vector l i =
[ l 1 i , l 2 i , . . .
, l ci ] (cid:62) ∈ { 0 , 1 } c to represent whether j -th label is the observed positive label, i.e., l ji = 1 if j = γ i , otherwise l ji = 0 .
The multi-label vector is denoted by y i =
[ y 1 i , y 2 i , . . .
, y ci ] (cid:62) ∈ { 0 , 1 } c , where y ji = 1 if the j -th label is relevant to x i and y ji =
if the j -th label is irrelevant.
The task of SPMLL is to induce a multi-label classiﬁer f : X (cid:55)→ 2 Y from (cid:101) D , which can assign a set of relevant label set for the unseen instance.
Recent work [5] empirically validates that SPMLL would reduce the amount of supervision with a tolerable damage in classiﬁcation performance.
The intuitive solution AN is assuming that unobserved labels are negative, which leads to the drawback that introduces some number of false negative labels.
Therefore, the SOTA approaches [5] aim to reduce the damaging effects of false-negative labels via employing the learning power of DNNs to achieve good performance in practice.
However, there is no existing method that can provide theoretical insights.
The Proposed Method 4.1 Risk-Consistent Estimator
To deal with single-positive multi-label learning, the classiﬁcation risk R ( f ) in Eq.
(1) could be rewritten as
[Equation]
Additionally, we employ the widely used loss function in multi-label learning, i.e, binary cross-entropy loss, as the loss function L ( f ( x ) , Y ) :
[Equation]
where (cid:96) j = log f
j ( x ) and ¯ (cid:96) j = 1 − f j ( x ) .
Then, (cid:80) Y ∈C L ( f ( x ) , Y ) p ( Y | x ) in Eq.
(2) could be calculated as
[Equation]
Here, d j = p ( y j = 1 | x ) ∈ [0 , 1] would be regarded as the soft label corresponding to class j for x .
By substituting Eq.
(4) into Eq.
(2), we obtain the following risk-consistent estimator for SPMLL
[Equation]
Therefore, we could express the empirical risk estimator via
[Equation]
Then, we could design a benchmark solution via applying the sigmoid function on f γ i ( x i ) to approximate p ( y γ i =
| x i ) and recovering the soft label d ji corresponding to each example via the label enhancement process in the following subsection.
4.2 Training with Label Enhancement
To recover the soft label vector d i =
[ d 1 i , d 2 i , . . .
d c i ] (cid:62) ∈ [0 , 1] c , S MILE considers the topological information of the feature space and estimates adjacency matrix A =
[ a ij ] n × n with
[Equation]
if
[Equation]
∈ [Equation] j ) [Equation]
otherwise
[Equation]
where N ( x j ) is the set for k -nearest neighbors of x j .
We assume that the latent soft label matrix D =
[ d 1 , d 2 , . . .
, d n ] generates the observed logical label matrix L =
[ l 1 , l 2 , . . . , l n ] and the adjacency matrix A . Besides, the observed instance matrix X =
[ x 1 , x 2 , . . .
, x n ] is generated from D and the latent feature matrix Z =
[ z 1 , z 2 , . . .
, z n ] .
We assume that the prior density p ( d ) is a Beta density with the minor values ˆ α =
[ˆ α 1 , ˆ α 2 , . .
, ˆ α c ] and ˆ β =
[ˆ β 1 , ˆ β 2 , . .
, ˆ β c ] , i.e., p ( d ) = (cid:81) cj =1 Beta (cid:16)
d j | ˆ α j , ˆ β j (cid:17) .
Then the prior density p ( D ) could be the product of each p ( d ) .
In addition, We assume that the prior density p ( z ) is a standard
Gaussian and prior density p ( Z ) can be represented as the product of each Gaussian p ( Z ) =
(cid:81) ni
=1 Gau ( z i | 0 , 1 ) . Then, the posterior density p ( D , Z | L , X , A ) can be decomposed as follows:
[Equation]
where L and A can be removed from the condition of p ( Z | D , L , X , A ) because of the independence between Z and L , A when latent variable D is given in the condition. Here we employ q ( D | L , X , A ) and q ( Z | D , X ) to approximate the true posterior p ( D | L , X , A ) and p ( Z | D , X ) respectively.
The approximate posterior q ( D | L , X , A ) could be the product of Beta parameterized by α i
=
[ α 1 i , α 2 i , . .
, α ci ] (cid:62) and β i =
[ β 1 i , β 2 i , . . .
, β ci ] (cid:62) :
[Equation]
Here, the parameters ∆ =
[ α 1 , α 2 , . . . , α n ] and Φ =
[ β 1 , β 2 , . .
, β n ] are the outputs of the inference model parameterized by w 1 as a GCN [19] with adjacency matrix by A .
Let q w {+
+} ( Z | D , X ) be the product of Gaussian parameterized by the mean vector µ i and standard deviation vector σ i :
[Equation]
The parameters Λ = [ µ 1 , µ 2 , . . . , µ n , σ 1 , σ 2 , . . .
, σ n ] are the outputs of the inference model with a MLP parameterized by w 2 .
We derive the evidence lower bound (ELBO) [18] on the marginal likelihood of the model to ensure that q w ( D , Z | L , X , A ) is as close as possible to p ( D , Z | L , X , A ) 1 :
[Equation]
We further assume that p ( X | D , Z ) is a product of each Gaussian with means ξ i and p ( L | D ) is a product of each multivariate Bernoulli with probabilities τ i .
In order to simplify the observation model, T ( m ) =
[ τ ( m ), τ ( m ), . . .
, τ ( m ) n ] is computed from m -th sampling D ( m ) with a MLP parameterized by η 1 and Ξ ( m ) =
[ ξ ( m ), ξ ( m ), . . . , ξ ( m ) n ] is computed from m -th sampling D ( m ) and Z ( m ) with a
MLP parameterized by η 2 .
Then the ﬁrst part of Eq.
(11) can be tractable as
[Equation]
Here, S ( · ) is the logistic sigmoid function, and implicit reparameterization trick [10] and MC sampling [18, 37, 38] are employed.
The second part of Eq.
(11) can be analytically calculated as
[Equation]
Here, Γ( · ) and ψ ( · ) are Gamma function and Digamma function, respectively.
The third part of Eq. (11) can be analytically calculated as follows: [Equation] Besides, we could promote the label enhancement process via enforcing that the estimated D should inherit the labeling-information of observed labels:
[Equation]
Finally, the objective of label enhancement T LE is obtained:
[Equation]
[Equation]
where λ is a hyper-parameter.
S MILE ﬁrst initializes the predictive network by warm-up training with AN solution, which would attain a ﬁne network before it starts ﬁtting noise [43].
Then we could sample the soft label from ﬁxed Beta after label enhancement and the sigmoid function on f γ i ( x i ) to approximate p ( y γ i = 1 | x i ) to make Eq.
(6) accessible, and train the predictive model θ by minimizing the risk estimator.
In each epoch, S MILE alternately operates label enhancement process and classiﬁer training process.
Algorithm 1 shows the algorithmic description of S MILE .
4.3 Estimation Error Bound
In this subsection, we establish an estimation error bound of the proposed method.
The empirical risk estimator according to Eq.(6) can be rewritten as:
[Equation]
17) where w ji = d ji p ( y γ =1 | x i ) c and ¯ w ji = 1 − d ji p ( y γ =1 | x i ) c .
Then the loss function L sp is
[Equation] [Table]
We deﬁne a function space as:
[Equation]
and denote the expected Rademacher complexity [1] of G sp as:
[Equation]
where σσσ = { σ 1 , σ 2 , . .
, σ n } is n Rademacher variables with σ i independently uniform variable taking value in { +1 , − 1 } .
Then we have
Lemma {+
+}
We suppose that the SPMLL loss function L sp could be bounded by M , i.e., M = sup x ∈X ,f ∈F ,y ∈Y L sp ( f ( x ) , y ) , and for any δ > 0 , with probability at least 1 − δ , then we have
[Equation]
The proof of Lemma 1 could be founded in Appendix A.3.
Lemma {+
 We suppose that+} the loss function (cid:96) ( f ( x ) , y ) and ¯ (cid:96) ( f ( x ) , y ) are ρ + -Lipschitz and ρ − - Lipschitz with respect to f ( x ) (0 < ρ + < ∞ and 0 < ρ − < ∞ ) for all y ∈ Y , respectively, and w j and ¯ w j are both bounded in [0 , κ ] .
Then, we have
[Equation]
where H y = { h : x (cid:55)→ f y ( x ) | f ∈ F} and R n ( H y ) =
E x ,σσσ (cid:104) sup h ∈H y 1 n (cid:80) ni
=1 h ( x i ) (cid:105) .
The proof of Lemma 2 could be founded in Appendix A.4.
Based one Lemma 1 and 2, we could obtain the following theorem:
Theorem 1 Assume the loss function (cid:96) ( f ( x ) , y ) and ¯ (cid:96) ( f ( x ) , y ) are ρ + -Lipschitz and ρ − -Lipschitz with respect to f ( x ) (0 < ρ + < ∞ and 0 < ρ − < ∞ ) for all y ∈ Y and the loss function L sp are bounded by M , i.e., M = sup x ∈X ,f ∈F ,y ∈Y L sp ( f ( x ) , y ) , with probability at least 1 − δ ,
[Equation]
Here, (cid:98) f sp = min f ∈F (cid:98) R sp ( f ) and f (cid:63) = min f ∈F R ( f ) are the empirical risk minimizer and the true risk minimizer, respectively.
The proof could be founded in Appendix A.5.
Theorem 1 shows that f sp would converge to f (cid:63) as n → ∞ and R n ( H y ) → 0 . [Table] [Table]
Experiments 5.1 Experimental Conﬁgurations
In the experiments, we adopt twelve widely-used MLL datasets [13], which cover a broad range of cases with diversiﬁed multi-label properties.
To evaluate the performance of SPMLL methods, we generate the single positive training data by randomly selecting one positive label to keep for each training example in the MLL datasets.
For each dataset, we run the comparing methods with 80%/10%/10% train/validation/test split.
The validation and test sets are always fully labeled.
The detailed descriptions of these datasets are provided in Appendix A.6.
Five popular multi-label metrics
Ranking loss , Hamming loss , One-error , Coverage , and Average precision [46] are employed for performance evaluation.
Furthermore, for Average precision , the larger the values the better the performance.
While for the other four metrics, the smaller the values the better the performance.
In this paper, S MILE is compared against four well-established SPMLL approaches including 1) A N [5] which assumes unobserved labels are negative and employs binary entropy loss for the training examples with the modiﬁed labels, 2) A N - LS [5] which assumes unobserved labels are negative and employs label smoothing [28] to reduce the impact of the incorrect labels (i.e. those labels incorrectly assumed to be negative), 3) W AN [5] which reduces the impact of false negatives by employing the down-weight terms in the loss corresponding to negative labels, and 4) R OLE [5] which online estimates of the unobserved labels throughout training and encourages the classiﬁer predictions to match the estimated labels via binary entropy loss.
As SPMLL could be regarded as the hardest version of MLL with missing labels, three wellestablished approaches of MLL with missing labels (also termed as MLL with partial labels) are adopted as the comparing approaches including 1) G LOCAL [47] which exploits global and local label correlations simultaneously via learning a latent label representation in the missing label cases, 2) M LML [34] which recovers the full label assignment for each sample by enforcing consistency with available label assignments and smoothness of labels, and 3) D2 ML [24] which utilizes both local low-rank label structures and label discriminant information for learning from missing labels.
For all the DNN-based approaches (A N , A N - LS , W AN , R OLE and S MILE ), we adopt three-layer MLP as the predictive model for fair comparisons and use the Adam optimizer [17]. The mini-batch size and the number of epochs are set to 16 and 25, respectively.
The learning rate and weight decay are selected from { 10 − 4 , 10 − 3 , 10 − 2 } with a validation set. Other hyper-parameters for all the comparing methods are also selected based on the validation.
All the comparing methods run 5 trials (with 80%/10%/10% train/validation/test split) on each dataset. [Table] [Table] [Figure]
[Figure]
[Table] [Figure] [Table] 5.2 Experimental Results
Tables 1 and 2 report the results of all approaches on Average precision and One-error , respectively.
For each evaluation metric, “ ↑ ” indicates “the smaller the better” while “ ↓ ” indicates “the larger the better”.
The results on other metrics are similar and could be seen in Appendix A.6.
In addition,
Wilcoxon signed-ranks test [6] is employed to show whether S MILE has a signiﬁcant performance than other comparing approaches.
Table 3 reports the p -values for the corresponding tests and the statistical test results at 0.05 signiﬁcance level.
Table 3 shows that S MILE achieves superior performance against all the comparing approaches on all evaluation metrics (except on Rranking loss and Coverage where S MILE achieves comparable performance against W AN ).
The superior performance of S MILE provides a strong evidence for the effectiveness of risk-consistent estimator for SPMLL.
Tables 1 and 2 show that the performance advantage of S MILE over comparing approaches is stable under varying the number of class labels.
In summary, these experimental results clearly validate the effectiveness of S MILE .
5.3 Further Analysis
To show the helpfulness of label enhancement to S MILE , a vanilla variant of S MILE (named S MILESI ) is adopted.
Here, label enhancement is replaced by approximating d ji with the conﬁdence of the current model f j ( x i ) , which is a widely-used technique [9, 22] to approximate the soft label in weakly supervised learning.
Table 4 reports detailed experimental results in terms of Average precision , One-error , and Ranking loss , respectively.
The detailed experimental results in terms of other metrics are reported in Appendix A.6.
Besides, the performance of each approach with the number of epochs on scene is shown in Figure 2.
Wilcoxon signed-ranks test [6] in Table 5 shows that S MILE achieves superior performance against S MILE - SI on all evaluation metrics, which clearly validates the usefulness of label enhancement.
Figure 3 illustrates the estimated D converges with the number of epochs on delicious , which shows that the estimated soft label could converge efﬁciently.
Conclusion
In this paper we study single-positive multi-label learning and propose a novel approach S MILE .
We derive an unbiased risk estimator, which suggests that one positive label of each instance is sufﬁcient to train predictive models for multi-label learning, and design a benchmark solution via estimating the soft label corresponding to each example in a label enhancement process.
The effectiveness of the proposed method is validated on twelve corrupted MLL datasets.
Acknowledgments
This research was supported by the National Key Research & Development Plan of China (No.
2021ZD0114202), the National Science Foundation of China (62206050, and 62076063),
China Postdoctoral Science Foundation (2021M700023), Jiangsu Province Science Foundation for
Youths (BK20210220)