E XPLOITING C LASS A CTIVATION V ALUE FOR P ARTIAL -L ABEL L EARNING
A BSTRACT
Partial-label learning (PLL) solves the multi-class classiﬁcation problem, where each training instance is assigned a set of candidate labels that include the true label.
Recent advances showed that PLL can be compatible with deep neural networks, which achieved state-of-the-art performance.
However, most of the existing deep PLL methods focus on designing proper training objectives under various assumptions on the collected data, which may limit their performance when the collected data cannot satisfy the adopted assumptions.
In this paper, we propose to exploit the learned intrinsic representation of the model to identify the true label in the training process, which does not rely on any assumptions on the collected data.
We make two key contributions.
As the ﬁrst contribution, we empirically show that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in images, could surprisingly be utilized to make accurate predictions on selecting the true label from candidate labels.
Unfortunately, as CAM is conﬁned to image inputs with convolutional neural networks, we are yet unable to directly leverage CAM to address the PLL problem with general inputs and models.
Thus, as the second contribution, we propose the class activation value (CAV), which owns similar properties of CAM, while
CAV is versatile in various types of inputs and models.
Building upon CAV, we propose a novel method named CAV Learning (CAVL) that selects the true label by the class with the maximum CAV for model training.
Extensive experiments on various datasets demonstrate that our proposed CAVL method achieves stateof-the-art performance.
I NTRODUCTION
To liberate humans from exhaustive label annotation work, numerous researchers have dedicated themselves to investigating various weakly supervised learning (WSL) (Zhou, 2017) problems, including but not limited to noisy-label learning (Liu & Tao, 2015; Xia et al., 2020; Han et al., 2020), semi-supervised learning (Zhu & Goldberg, 2009; Miyato et al., 2018;
Luo et al., 2018), and multiple-instance learning (Zhou et al., 2012).
This paper focuses on another popular WSL problem called partial-label learning (PLL) (Jin & Ghahramani, 2002;
Cour et al., 2011), which aims to learn a model from training examples equipped with a set of candidate labels that include the true label.
Due to the cost and difﬁculty of annotating every example exactly with the true label in huge datasets, PLL has been widely applied to various tasks such as multimedia context analysis (Zeng et al., 2013) and web mining (Luo & Orabona, 2010).
To solve the PLL problem, there are two mainstream strategies to discriminate the unknown true label from candidate labels, including the average-based strategy (ABS) and the identiﬁcation-based strategy (IBS).
ABS always treats each candidate label equally and averages the model outputs of all candidate labels for prediction (H¨ullermeier & Beringer, 2006;
Cour et al., 2011;
Feng et al., 2020; Yao et al., 2020; Wen et al., 2021).
IBS concentrates on iteratively selecting one label from the candidate label set as the true label to exclude the uncertainty, forming PLL into an ordinary classiﬁcation problem (Jin & Ghahramani, 2002; Liu & Dietterich, 2014; Zhang & Yu, 2015; Zhang et al., 2016).
Among these methods, only several of them (Yao et al., 2020; Wen et al., 2021) can be compatible with deep neural networks, which achieved state-of-the-art performance.
Most of the existing PLL methods aim at designing proper training objectives under various assumptions on the collected data.
For example, the consistent methods proposed by Feng et al.
(2020) are based on a speciﬁc data generation assumption.
The derivation of the PRODEN method (Lv et al., 2020) stems from the assumption that the true label achieves the minimal loss among the candidate labels.
Yan & Guo (2021) proposed the MGPLL method based on the assumption of non-random label noise.
Although these methods have achieved generally great empirical performance, their performance could be degraded when the collected data cannot meet the adopted assumptions.
In this paper, we aim to investigate a novel PLL method by exploiting the learned intrinsic representation of the model to identify the ground truth in the training process without relying on any assumptions on the collected data.
We focus on the class activation map (CAM)
(Zhou et al., 2016), which is a prevailing tool for visualizing the representation information of convolutional neural network (CNN)-based models and could be easily obtained by the weighted linear sum of the feature maps.
As CAM is able to discriminate the learning patterns of each class for the model, we conjecture that CAM could be used to guide the model to spot the true label.
To verify this assumption, we conducted a pilot experiment on CIFAR-10 (Krizhevsky et al., 2009), where the candidate label sets were generated in two different ways.
As shown in Figure 1 (please refer to Section 2.2), it is surprising to ﬁnd that CAM can potentially help the model recognize the true labels, and such capacity of CAM synchronously changes with the classiﬁer performance during the whole training phase.
Based on the experimental results, it can be conjectured that such intrinsic representations can be useful to distinguish the true label from candidate labels for PLL.
Motivated by the above observations, we naturally consider leveraging CAM to solve the PLL problem.
However, CAM was only proposed for addressing image datasets with using deep models built on CNN, revealing two limitations on its application.
Firstly, CAM cannot be adopted to the classiﬁer based on shallow models such as the linear model and multilayer perceptron (MLP).
Secondly,
CAM is unable to deal with any inputs other than the image.
To overcome these shortcomings, we for the ﬁrst time propose a simple but effective tool—the class activation value (CAV) to capture the learned representation information in a more general way, which is essentially the weighted output of the classiﬁer.
Experimental results in Figure 2 and Appendix B.7 also shows that our CAV works similarly as CAM during the training phase.
Building upon CAV, we propose a simple yet effective PLL method called CAV Learning (CAVL) that guides the model to differentiate the true label from the candidate set during the training process.
Speciﬁcally, we ﬁrst train the model by treating all candidate labels equally and obtain CAVs of given training examples, and then regard the class with the maximum CAV as the true label for each instance during the training process.
In this way, CAVL transforms PLL into supervised learning, thereby making the model reliable to recognize the true label with learned representation information by CAV.
Extensive experiments on benchmark-simulated and real-world datasets show that our proposed CAVL method achieves state-of-the-art performance.
D ISCOVERING C LASS A CTIVATION M AP FOR P ARTIAL -L ABEL L EARNING
In this section, we provide a detailed discussion about our motivation and empirically show that the class activation map could be helpful for addressing PLL.
2. C LASS A CTIVATION M AP
The class activation map (CAM) (Zhou et al., 2016) is a popular and elementary mechanism in computer vision to represent the discriminative part of an input image captured by the classiﬁer to identify each class.
In other words, CAM manifests the learning patterns of the classiﬁer, denoted as f , to the speciﬁc class in the image.
Let us denote an input image as x ∈ R d = c × h × w (with c channels, height h and width w ) and CAM of x ∈ R d = c × h × w as m ∈
R k × h × w where k is the number of classes.
We note that m could be generated by training a speciﬁc classiﬁcation model f c , which normally comprises of an encoder function e : X → R c f × h × w for feature map extraction, a
Global Average-Pooling layer gap : R c f × h × w → R c f × 1 , and a linear layer with weight θ ∈ R k × C f , where c f is the number of channels in the feature maps.
Therefore, CAM of x related to the j -th class could be obtained by
[Equation]
where θ j ∈ R 1 × C f is the corresponding linear weight related to the j -th object.
CAM is a weighted linear sum of feature maps and the linear weights.
The weakness of CAM is the limited applications, since not all classiﬁcation network architectures follow the model f c mentioned above.
To improve its generalization for any CNN-based model f , Gradient-weighted CAM (Grad-CAM) (Selvaraju et al., 2017) was proposed to implement such an internal representation by reasonably leveraging class-speciﬁc gradient information.
Concretely, let us denote by g j ∈ R d = c × h × w the Grad-CAM of x related to the j -th object, and g j could be expressed as
[Equation]
Intuitively, the derivative of logit f ( x ) with respect to feature map e ( x ) is used as the weights for calculating g .
Note that both Grad-CAM and CAM are generated by weighted sums of the feature maps, and Grad-CAM is equivalent to CAM when f follows the same architecture as f c .
Based on
Grad-CAM, Grad-CAM++ (Chattopadhay et al., 2018) aims to provide better visual explanations of
CNN and occurrences of multiple foreground objects.
Thanks to such a simple technique, numerous problems in computer vision such as interpretation (Xu et al., 2019a) and weakly supervised semantic segmentation (Wei et al., 2016) have achieved marvelous progress.
In this paper, we aim to extract and improve the useful knowledge from CAM to address the PLL problem, leading the classiﬁer to identify the ground truth.
2. P ILOT EXPERIMENT ON CAM
Generally, Grad-CAM or CAM is treated as the internal representation of f .
We believe that such a mechanism could guide f to differentiate the true label from the candidate set because it is constructed by taking advantage of internal elements in f .
To validate our conjecture, we conducted a pilot experiment by adopting two different label selection methods.
The ﬁrst one is the intuitive method (IM), which simply regards all the labels from the candidate sets as the true labels with using cross entropy loss.
The second one is the “CAM label” selection strategy, which discriminates the true label from the candidate label set by using CAM.
During each training epoch, we calculated the number of the foreground seeds of CAM from each candidate label set.
Speciﬁcally, to calculate the number of foreground seeds of m j , we counted the elements with positive values in m j .
Note that each x could possess k CAMs and the true label is always in the candidate set.
Thus we selected the label from the candidate set, whose CAM owns most foreground seeds as the true label.
The label candidate label sets were generated by two different approaches: (I) Uniformly Sampling
Strategy (USS).
Uniformly sampling the candidate label set for each training instance from all the possible candidate label sets (Feng et al., 2020).
(II) Flipping Probability Strategy (FPS).
By setting a ﬂipping probability q to any false label, the false label could be selected as a candidate label with a probability q
(Feng & An, 2019a; Yan & Guo, 2020;
Lv et al., 2020; Wen et al., 2021).
Here the detailed training settings of the experiments are similar to the experiment part (please refer to
Section 4.1.1 for details).
ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) were chosen as the backbones to train a classiﬁer on CIFAR-10 (Krizhevsky et al., 2009).
Figure 1 shows the average results of 5-time trials and we can ﬁnd that CAM demonstrates two helpful features during the training phase.
Firstly, it is clear that the performance of the classiﬁer [Figure] trained by “CAM label” selection strategy is better than IM (the gap between the blue dashed line and the yellow dashed line), which shows that CAM could potentially guide the classiﬁer toward the true label from the candidate set.
Here we name this attribute of CAM as power .
Secondly, it is found that the accuracy of identifying true labels by CAM is synchronously changed with that by the classiﬁer.
Several ﬂuctuations (marked by the paired circle) of the classiﬁer performance (test accuracy) and CAM accuracy are similar to a large extent, resulting in continuous improvement to the model itself.
Here we name this observed feature of CAM as dynamic .
The power and dynamic attributes disclose the fact that CAM may show more than we thought , which is an inspiring ﬁnding to address PLL.
The power attribute motivates us to consider that the classiﬁer may learn more accurate true labels if it is forced to approximate the label distribution recognized by CAM.
The dynamic attribute may guarantee that such guidance would be constantly effective during the whole training phase since CAM would synchronously be self-improved as the classiﬁer becomes stronger.
Hence it would be reasonable and meaningful to explore such internal sign as guidance for PLL.
P ROPOSING C LASS A CTIVATION V ALUE FOR P ARTIAL -L ABEL L EARNING 3. C LASS A CTIVATION V ALUE —S IMPLE AND GENERAL REPRESENTATION
In this paper, we aim to propose a CAM-like tool as guidance for f to identify the true label from candidate labels.
CAM or Grad-CAM is specially designed for convolutional neural network (CNN)- based models, which mainly deals with image datasets with retaining spatial information.
However, to the best of our knowledge, most PLL methods are implemented in datasets with various forms, including the image datasets such as MNIST (LeCun et al., 1998), and feature vectors such as BirdSong (Briggs et al., 2012).
The networks on these datasets could be directly adapted to elementary models such as the linear or multilayer perceptron (MLP) for f to deal with the inputting ﬂatteneddimension vectors.
Without spatial information or CNN-based architectures, CAM-based tools are invalid to represent the internal learning pattern.
To make such a concept versatile, we aim to propose another novel CAM-like tool with simple implementation that could be adopted in a wider range of backbones and problems.
We note that the (softmax) outputs can replace the feature map e ( x ) since the encoder e could be directly regarded as f .
Thus, it is fundamental to formulate an internal component for a value-based framework.
For each candidate label, apart from its class output, its gradient of training loss with respect to the model output can also capture the importance degree of the candidate label to some extent.
Inspired by the formulation of Grad-CAM in Eq.
(2) that regards the gradient information ﬂow to the last convolution layer as the importance of each neuron to the j -th class, we can intuitively derive the importance of each candidate label as follows: [Equation]
[Equation]
where ψ : R k
→ R k is the activation function imposed on the logit vector f ( x ) to approximate the class-conditional probability P ( y | x ) , and ψ is mostly implemented by the SoftMax function.
The derivation process of the last inequality of Eq.
(3) is provided in Appendix A. From Eq.
(3), we can observe that the importance v j of each candidate label j depends on the softmax conﬁdence score ψ j ( f ( x )) .
However, we argue that such a calculation of v j is actually suboptimal, because ψ j ( f ( x )) conveys less information of model output than the logit f j ( x ) as ψ j ( f ( x )) could be considered as a condensed version of f j ( x ) .
Therefore, instead of directly adopting Eq.
(3), we propose to replace ψ j ( f ( x )) in Eq.
(3) by f j ( x ) , and the importance of each candidate label would be calculated by the following simple way:
[Equation]
We call the importance v j calculated by Eq.
(4) as the class activation value (CAV) of the j -th class, as it could be regarded as a value-based “CAM”.
Compared with CAM, we directly treat f ( x ) as the “feature map” so that v can be obtained by a weight imposed on the class output of the model.
In summary, both CAV and CAM are the weighted operation to the feature maps (or outputs in the classiﬁer).
Differently from CAM, CAV is essentially a value so that it could be easily obtained and applied to different backbones.
Besides, our CAV could also be generated with less time complexity since there is no need to process the weighted sum of the feature maps.
3. P ARTIAL -L ABEL L EARNING WITH C LASS A CTIVATION V ALUE
In this section, we introduce how to effectively address PLL by using CAV in detail.
Firstly, we give a formal introduction to PLL.
For a k -class classiﬁcation problem, let X ∈ R
d be the feature (input) space and { x i } mi =1 ∈
X be the input data, where m refers to the number of training examples.
Let Y = { 1 , 2 , ..., k } be the label space.
Let us denote C
= { 2 Y \∅\ Y } as the candidate label space where 2 Y is the power set of Y , and | C | = 2 k − 2 shows that the candidate label set is not supposed to be the empty set nor the whole label set.
For each training instance x i , let y i ∈
Y be the ground truth label and
S i ∈
C be the candidate label set.
We denote P ( x , y ) and P ( x , S ) as the probability densities of fully labeled examples and partially labeled examples respectively.
Based on the crucial assumption of PLL that the candidate label set of each instance must include the correct label, we have y i ∈ S
i , i.e.,
[Equation]
PLL aims to learn a classiﬁer f : X → R k with training examples independently and identically sampled from P ( x , S ) to make correct predictions for test examples.
To the best of our knowledge, most methods focus on designing the loss by considering all candidate labels or iteratively extracting the true label from the theoretical aspect.
Differently from them, we concentrate on how to solve
PLL by efﬁciently utilizing the learned representation by CAV, which could be potential and effective guidance for helping f recognize the true label.
Note that the learned representation could be directly obtained by using CAV after ﬁrstly training f in one epoch.
For training f in the ﬁrst epoch, here we directly utilize the IM strategy, i.e., treating every candidate label equally by cross entropy loss L ( f ( x ) , s ) , s ∈ S .
Hence the empirical risk function ˆ R 1 for the ﬁrst epoch could be deﬁned as
[Equation]
After training f in one epoch, the CAV v of every candidate label could be gained for each x by
Eq.
To keep the classiﬁer away from the negative effect by the false positive labels from the candidate set, we use the knowledge by CAV to guide the following learning process after the ﬁrst training epoch, and the potential true label for x i is selected by
[Equation]
where y (cid:48)
i is the selected true label from the candidate label set for x i .
In this way, y (cid:48)
i could be straightly treated as the ground truth so that PLL could be transferred to classical supervised learning.
Finally, the risk function ˆ R cavl for the following epochs could be expressed as
[Equation]
As f gradually learns to approximate P ( y (cid:48) | x ) , CAV would also be updated to identify the true label.
Based on such a progressive label selection on the candidate label set, our Class Activation Value
Learning (CAVL) is supposed to progressively training an accurate model with correctly selected labels.
The pseudo-code of CAVL is given in Algorithm 1.
E XPERIMENTS
In this section, extensive experiments on various datasets are implemented to verify the effectiveness and rightness of our proposed CAV and CAVL method.
4. CAVL PERFORMANCE 4.1. B ENCHMARK DATASET COMPARISONS
Datasets and backbones.
We use four popular benchmark datasets to test the performance of our
CAVL, which are MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), KuzushijiMNIST (Clanuwat et al., 2018) and CIFAR-10 (Krizhevsky et al., 2009).
Note that it is necessary to manually generate the candidate label sets since they are supposed to be used for single-classiﬁcation problems.
Recall that we introduce two different candidate label generation process (please refer to
Section 2.2 for details)
(Feng & An, 2019a;b; Feng et al., 2020; Lv et al., 2020; Wen et al., 2021), i.e., USS and FPS.
The former one is implemented by uniformly sampling a label set from all the partial label space C for each instance, and the latter one sets a ﬂipping probability q to any irrelevant label so that it could become one item in the candidate label set with probability q .
In this section we set q ∈ { 0 . 3 , 0 .
7 } to represent different ambiguity degrees.
For MNIST, Kuzushiji-MNIST and Fashion-MNIST, we adopt 3-layer MLP and 5-layer LeNet as the backbones.
For CIFAR-10, we choose 34-layer ResNet (He et al., 2016) and 22-layer DenseNet (Huang et al., 2017) as the base models.
Compared methods and training settings.
We compare our CAVL with four state-of-the-art approaches including PRODEN (Lv et al., 2020), Leveraged Weighted (LW) Loss (Wen et al., 2021), RC and CC (Feng et al., 2020).
For all methods we search the initial learning rate from { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 } , and weight decay from { 10 − 6 , 10 − 5 , ..., 10 − 1 } .
We take a mini-batch size of { 32 , 256 } images and train all the methods using Adam (Kingma & Ba, 2015) optimizer for 250 epochs.
Hyper-parameters are searched to maximize the accuracy on a validation [Table] [Table] set containing 10% of the training samples annotated by true labels.
All the implemented methods are trained on 1 RTX3090 GPU with 24 GB memory.
To guarantee the fairness of comparisons, we repeatedly conduct all experiments 5 times and record the average accuracy with the standard deviation following the conventions in Feng et al.
Lv et al.
Wen et al.
This work is partially supported by Huawei MindSpore (Huawei, 2020).
Experiment results.
As shown in Table 1 and 2, our CAVL shows its superiority on four benchmark datasets with various backbones.
Additionally, our CAVL outperforms nearly all state-of-the-art methods using the candidate labels generated by both USS and FPS, which shows that our CAVL does not rely on any data generation assumption.
Speciﬁcally, it is worth mentioning that CAVL is able to show competitive performance with RC, which is theoretically proved to be possess optimal [Table] performance in data distribution generated by USS.
Therefore, the results reasonably verify the generalization and effectiveness of
4.1. R EAL - WORLD DATASET COMPARISONS
Datasets and backbones.
We select ﬁve real-world datasets including Lost (Cour et al., 2011),
MSRCv2 (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng et al., 2013) and Yahoo!News (Guillaumin et al., 2010).
Note that these real-world datasets have their candidate labels.
As a common practice in (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021), we select 1-layer linear model and 3-layer MLP as the backbones to validate the performance of our CAVL in these ﬁve datasets.
Compared methods and training settings.
Apart from the four methods mentioned in Section 4.1.1, we add SURE (Feng & An, 2019b), CLPL (Cour et al., 2011), IPAL (Zhang & Yu, 2015), PLSVM (Elkan & Noto, 2008), PLECOC (Zhang et al., 2017) and PLKNN (H¨ullermeier &
Beringer, 2006) as extra methods for comparison, where the parameters are searched to maximize the accuracy on a validation set (10% of the training set).
We repeatedly run all experimentstimes, and report the average accuracy with the standard deviation for each method.
Other settings are similar to Section 4.1.1.
The comparison results using MLP is recorded in Appendix B.2.
Experiment results.
In Table 3, our proposed CAVL shows the best performance in nearly all ﬁve real datasets.
Besides, CAM is unable to be extracted with the word embedding input such as
BirdSong, lacking spatial information.
Therefore, the achievement of CAVL on these real datasets shows the generalization and superiority of our CAVL to shallow model, where CAV could serve as the CAM to lead the classiﬁer to learn the true label.
4. CAV ATTRIBUTES
In this part, we conduct several experiments to present how these attributes of CAV guide the learning process.
We implement CAVL with four backbones including linear, MLP, ResNet and DenseNet on four benchmark datasets, whose candidate label sets are generated by USS.
The training settings are similar to the ones in Section 4.1.1.
Figure 2 illustrates the averaging performance of CAV, CAM (for deep networks) and the model itself during the 5-time training process.
The red (blue) line in Figure 2 depicts the training accuracy measured by treating the maximum CAV (CAM) from the candidate set as the true label, and the cyan line depicts the test accuracy of the model.
We note that here CAM is simply extracted from the classiﬁer for comparison, which is different from its usage in Section 2.2.
As shown in Figure 2, it is clear that the classiﬁer learns to approximate the better data distribution by CAV, and the accuracy trend recognized by CAV is consistent with the classiﬁer performance in the test datasets, which validates the Power and Dynamic attribute of CAV.
R ELATED W ORK
In this section, we give a brief introduction to the two mainstream strategies for partial-label learning (PLL), i.e., the averaged-based strategy (ABS) and the identiﬁcation-based strategy (IBS).
ABS treats all candidate labels in an equal manner and then averages the model outputs of all candidate labels for prediction.
Some non-parametric methods (H ¨ ullermeier & Beringer, 2006; Gong et al., 2017) concentrated on predicting the label by leveraging the outputs of its neighbors.
Furthermore, some methods (Cour et al., 2009; Zhang et al., 2016; Yao et al., 2020) aimed to subtly leverage the labels outside the candidate set so as to discriminate the potential true label.
Yao et al.
(2020) designed an entropy-based regularizer to minimize the entropy of each label to maximize the margin between the potential true label and the unlikely labels.
A recent study (Feng et al., 2020; Lv et al., 2020; Wen et al., 2021; Lv et al., 2021) focused on the data generation process and proposed a classiﬁer-consistent method based on a transition matrix.
(Chen et al., 2020) proposed a new problem General Partial Label Learning and leveraged graph neural network to tackle the label ambiguity.
IBS aims to constantly identify the most possible true label from the candidate label set to eliminate the label ambiguity.
Early works treated the potential truth label as a latent variable and optimized the objective function by the maximum likelihood criterion (Jin & Ghahramani, 2002; Liu & Dietterich, 2014) or the maximum margin criterion (Nguyen & Caruana, 2008; Yu & Zhang, 2016).
Later, taking advantage of topological information of feature space to generate the score of each candidate label (Zhang & Yu, 2015; Zhang et al., 2016;
Feng & An, 2018; Wang et al., 2019) attracted much attention from researchers.
These methods commonly aimed to iteratively update the conﬁdence of each candidate label based on the widely used assumption that similar instance are supposed to possess the same label.
Xu et al.
(2019b) proposed to model the generalized label distribution by leveraging the topological information of the feature space.
Lyu et al.
(2021) reformulated PLL into a matching selection problem and proposed a Graph-Matching based Partial Label
Learning framework to solve the problem.
It is worth noting that IBS is commonly susceptible to the false positive labels that co-occur with the true label in the candidate label set.
Thus our CAVL, which belongs to IBS, also owns this inevitable shortcoming.
C ONCLUSION
In this paper, we exploited the learned intrinsic representation of the model for partial-label learning (PLL).
We made two key contributions.
Firstly, we found that the class activation map (CAM), a simple technique for discriminating the learning patterns of each class in an image, could surprisingly be used to differentiate the true label from candidate labels.
Unfortunately, we are yet unable to directly use CAM for PLL, as CAM is subject to image inputs with convolutional neural networks.
Thus, our second contribution is to propose the class activation value (CAV), which not only owns similar properties of CAM but also is versatile in various types of inputs and models.
Based on CAV, we proposed a simple yet effective PLL method that selects the true label by the class with the maximum CAV for model training.
Comprehensive experimental results on various datasets demonstrated that our CAVL achieved state-of-the-art performance. ACKNOWLEDGMENT
FZ and BH were supported by NSFC Young Scientists Fund No. 62006202, RGC Early Career Scheme No. 22200720 and HKBU CSD Departmental Incentive Grant. BH was also supported by MSRA StarTrack Program. LF was supported by the National Natural Science Foundation of
China (Grant No. 62106028) and CAAI-Huawei MindSpore Open Fund.
TLL was supported by
Australian Research Council Projects DE-190101473 and DP-220102121.
MS was supported by
JST CREST Grant Number JPMJCR18A2.
A D ERIVATION PROCESS OF E QUATION . (3)
The SoftMax function is ψ j ( x ) = exp( x j ) / (cid:80) kj =1 exp( x j ) , where x ∈ R d refers to the.
Then we can obtain the derivation of − log( ψ j ( x )) with respect to the x j by
[Equation]
B D ETAILED SUPPLEMENTARY FOR E XPERIMENTS B. B ENCHMARK D ATASETS
In Section 4.1.1, we use four widely-used benchmark datasets, i.e. MNIST (LeCun et al., 1998),
Fashion-MNIST (Xiao et al., 2017), Kuzushiji-MNIST (Clanuwat et al., 2018) and CIFAR-
(Krizhevsky et al., 2009).
Table 4 lists the characteristics of these datasets.
We respectively describe these datasets as follows.
• MNIST:
It is a 10-class dataset of handwritten digits.
Each data is a 28 × 28 grayscale image.
• Fashion-MNIST:
It is also a 10-class dataset.
Each instance is a fashion item from one of the 10 classes, which are T-shirt/top, trouser, pullover, dress, sandal, coat, shirt, sneaker, bag, and ankle boot.
Moreover, each image is a 28 × 28 grayscale image.
• Kuzushiji-MNIST:
Each instance is a 28 × 28 grayscale image associated with one label of 10-class cursive Japanese (“Kuzushiji”) characters.
• CIFAR-10:
Each instance is a 32 × 32 × 3 colored image in RGB format.
It is a ten-class dataset of objects including airplane, bird, automobile, cat, deer, frog, dog, horse, ship, and truck.
B. R EAL D ATASETS
In Section 4.1.2, We select ﬁve real-world datasets including Lost (Cour et al., 2011), MSRCv (Liu & Dietterich, 2012), BirdSong (Briggs et al., 2012), Soccer Player (Zeng et al., 2013) and
Yahoo!News (Guillaumin et al., 2010).
Here we make a comprehensive descriptions about them shown as follows.
Lost, Soccer Player and Yahoo!
News: They crop faces in images or video frames as instances, and the names appearing on the corresponding captions or subtitles are considered as candidate labels.
• MSRCv2: Each image segment is treated as a sample, and objects appearing in the same image are regarded as candidate labels.
• BirdSong:
The singing syllables of birds are regarded as instances and bird species who are jointly singing during any ten seconds are represented as candidate labels
[Table] [Table] [Table] Table 5 shows the average number of candidate labels (Avg.
# CLs) per instance.
Table 6 presents the performance comparison between other methods and our CAVL method based on MLP model, the results of which also validate the effectiveness our CAVL.
B. A BLATION STUDIES ON THE STARTING EPOCH IN CAVL
In Section 3.2, we propose the CAVL method to address PLL.
Speciﬁcally, we use the IM to train the classiﬁer in the ﬁrst epoch, and then the potential true labels could be obtained with CAV after the following training epochs.
Here we set the epoch number for starting using the CAVL method, i.e. the starting epoch, as one.
To further validate the effectiveness of our CAVL, we explore the effects of the starting epoch on our method.
Speciﬁcally, we implement CAVL with two backbones including LeNet and ResNet on four benchmark datasets.
We repeatedly run the experiments 5 times by using candidate labels generated by USS.
Following similar training settings in Section 4.1.1, we set the total training epoch as 250, and select the starting epoch from [10 , 50 , 100 , 150] .
IM is used before the selected starting epoch and CAVL is implemented after the selected epoch.
As shown in Table 7, the accuracy of the classiﬁer becomes lower as the increase of the starting epoch in CAVL.
Let us take the results in KMINIST as an example, the CAVL shows its superiority with setting the starting epoch as 1, achieving 93 .
25 ± 0 .
21% accuracy performance.
As the starting epoch increases, the classiﬁer is more negatively affected by the false positive labels in the candidate sets and dragged away from the true labels by IM.
Thus it only achieves 88 .
54 ± 0 .
26% with the starting epoch as 150.
This phenomenon is reasonable because the DNN learn patterns ﬁrst, which suggests that deep networks can gradually memorize the data, moving from regular data to irregular data such as outliers and mislabeled data (Arpit et al., 2017).
Therefore, our CAVL could not [Table] sufﬁciently help the classiﬁer to deal with this memorization effect during the rest of the training epochs.
In conclusion, the effectiveness of our CAVL would be weakened as extending the starting epoch
[Table]
B. G ENERATION OF C ANDIDATE LABELS
In Section 4.1.1 we introduce two different generation ways for the candidate label sets [Table] i [Table]
e, USS,
Uniformly sampling a label set from all the partial label space C for each instance.
FPS, Setting a ﬂipping probability q to any irrelevant label which could possibly become one item in the candidate label set with probability q .
For USS, each partially labeled example ( x , S ) is independently drawn from a probability distribution with the following density:
[Equation]
The generation process assumes that the candidate label set S is independent of the instance x .
There are totally 2 k − 1 possible candidate label sets that contain the speciﬁc true label y .
Therefore,
Eq.
(10) illustrates the candidate label set for each instance is uniformly sampled.
For FPS, we set a ﬂipping probability q to any irrelevant label that possibly entries the candidate label set.
Here we introduce the class transition matrix (denoted by T ) for partially labeled data, where T ij refers to the probability of the label j being a candidate label given the true label i for each instance.
Note that T ii = 1 always holds since the true label always belongs to the candidate label.
T ij = q, i (cid:54) =
j holds for other elements.
The matrix representation of T is expressed as:    q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q   
B. C OMPARED M ETHODS
The compared PLL methods are listed as follows.
• SURE (Feng & An, 2019b):
It iteratively enlarges the conﬁdence of the candidate label with the highest probability to be the correct label.
• CLPL
(Cour et al., 2011): It uses a convex formulation by using the one-versus-all strategy in the multi class loss function.
• IPAL (Zhang & Yu, 2015):
It is a non-parametric method that applies the label propagation strategy to iteratively update the conﬁdence of each candidate label.
• PLSVM (Elkan & Noto, 2008):
It is a maximum margin-based method that differentiates candidate labels from non-candidate labels by maximizing the margin between them.
• PLECOC (Zhang et al., 2017):
It adapts the Error-Correcting Output Codes method to deal with partially labeled examples in a disambiguation-free manner.
• PLKNN (H¨ullermeier & Beringer, 2006):
It adapts the widely-used k-nearest neighbors method to make predictions for partially labeled examples.
• CC & RC (Feng et al., 2020):
The former method is a novel risk-consistent partial label learning method and the latter one is classiﬁer-consistent based on the generation model.
• LW (Wen et al., 2021):
The method proposes a family of loss function for the ﬁrst time, where introduces the leverage parameter β to consider the trade-off between losses on partial labels and non-partial labels.
For all the above methods, their parameters are speciﬁed or searched according to the suggested parameter settings by respective papers to maximize the accuracy performance on a validation set, which is made by the 10% of the training set.
B. D ETAILS OF THE BACKBONES
In Section 4 we select ﬁve network architectures as the backbones for modelling our classiﬁer, which are linear, 3-layer MLP, 5-layer LeNet, 34-layer ResNet (He et al., 2016) and 22-layer
DenseNet (Huang et al., 2017).
The linear model is a linear-in-input model: d − k .
MLP refers to a 3-layer fully connected networks with ReLU as the activation function, and the architecture is d − 500 − k .
LeNet is comprised of 2 Convolution Layer and 3 fully connected layer.
34-layer
ResNet and 22-layer DenseNet strictly follow the implementation by He et al.
and
Huang et al.
B. P
In Section 2.2 we present the pilot experiments on CIFAR10, which shows that CAM possess two attributes, i.e., power and dynamic .
Here we provide more results on ﬁve real-world datasets to prove the rightness of our proposed CAV.
The real-world datasets already have their candidate labels.
Following similar training settings in Section 2.2, we select 1-layer linear as the backbone for the classiﬁer and respectively use the IM and CAV to train it in 10 epochs.
Intuitively, using CAV to train the model is equal to CAVL.
Note that CAM could not be obtained in the linear module.
Figure 3 shows the average results of 10-time trials and it is easily found that CAV also shows power and dynamic attributes in real-world datasets, validating the generalization and effectiveness of our
CAV.
C F URTHER EXPLANATIONS ON C LASS A CTIVATION M AP
Here we provide a detailed explanation of CAM.
For a well-trained CNN module for addressing the classiﬁcation problem, the Class Activation Map (CAM) is simply a weighted linear sum of the presence of the visual patterns at different spatial locations for the input images.
CAM aims to mine out what the network focuses on the input images related to true labels during the training phase.
It is known that a CNN-based module is a ”black box” and the meaning behind CAM is the exploration of the inner principles inside such technology.
Based on CAM, it is convenient to visualize the learning patterns of the CNN-based modules.
Figure 4 shows several CAM results selected from (Zhou et al., 2016).
Here we take one example of the Teapot for illustration, the CAM of the image related to the teapot shows the most discriminative regions in the head part, which manifests that the classiﬁcation classiﬁes the image to the teapot since it ﬁnds and concentrates on the head part of the teapot.
Thanks to CAM, we could conveniently and effectively investigate the inner principles of CNN-based modules by simply utilizing the components from the network itself.
This is also the reason why we name it ”internal representation”, which is essentially a form of intermediate outputs.