SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Partial-label learning (PLL) is a peculiar weakly-supervised learning task wherethe training samples are generally associated with a set of candidate labels insteadof single ground truth.
While a variety of label disambiguation methods have beenproposed in this domain, they normally assume a class-balanced scenario that maynot hold in many real-world applications.
Empirically, we observe degeneratedperformance of the prior methods when facing the combinatorial challenge fromthe long-tailed distribution and partial-labeling.
In this work, we first identifythe major reasons that the prior work failed.
We subsequently propose SoLar , anovel Optimal Transport-based framework that allows to refine the disambiguatedlabels towards matching the marginal class prior distribution.
SoLar additionallyincorporates a new and systematic mechanism for estimating the long-tailed classprior distribution under the PLL setup.
Through extensive experiments, SoLarexhibits substantially superior results on standardized benchmarks compared to theprevious state-of-the-art PLL methods.
Introduction
The remarkable success of deep learning typically requires collecting massive labeled data, whichis notoriously labor-intensive.
Of particular concern, data labeling can suffer from inherent andpervasive label ambiguity.
Take Figure 1 (a) as an example, a Mule can be visually similar to both
Donkeys and Horses, which hinders non-expert annotators from recognizing the true breed.
Similarcases also arise in many real-world applications, such as automatic image annotation [1], bird songrecognition [2] and facial age estimation [3].
To deal with ambiguous supervision, partial-labellearning (PLL) [4, 5], which allows each training example to be annotated by a set of candidate labels,has attracted significant attention from the community.
A plethora of methods have been developedto tackle this problem, including pseudo-labeling methods [6, 7], graph-based methods [8, 9, 10] etc.
Despite the promise, existing PLL methods have been commonly driven by the assumption thattraining data consists of class-balanced distribution, which may not hold in practice.
In many realworld scenarios, training data exhibits a long-tailed label distribution [11].
That is, many labels occurinfrequently in the training data.
Concerningly, the imbalanced data can exacerbate the fundamentalchallenge of label disambiguation, i.e. , identifying the ground-truth label from candidates.
Indeed, wefind that current best-performing PLL methods display degenerated performance in the long-tailedsetting.
This happens because the predictions of pseudo-labeling—a core component that PLLmethods rely on—can be largely biased towards the head and majority classes.
We exemplify thephenomenon in Figure 1 (b), where a strong method PRODEN [6] rarely assigns probability mass tothe tail classes (indexed by 9 and 10).
As a result, these tail classes suffer from almost zero accuracy,as shown in Figure 1 (c).
To date, few efforts have been made to resolve this.
[Figure]
Motivated by this, we propose a novel framework for long-tailed partial-label learning, called
S inkh o rn La bel r efinery (dubbed SoLar ).
Our framework emphasizes the long-tailed nature oftraining data, while performing label disambiguation for partial-label learning.
Our key idea isto enforce constraints on pseudo-labels—the distribution of which should match the class priordistribution .
The constraints can incentivize the label disambiguation process to select tail labelsfrom candidate sets, instead of always selecting head classes.
We formalize the idea as an optimaltransport problem [12] that searches for a proper label assignment subjected to constraints: (1) theprobability mass is distributed within candidate sets; (2) the distribution of (predicted) pseudo-labelsmatches the prior class distribution.
The first constraint is natural in the PLL setup, and the latter oneuniquely empowers SoLar to disambiguate tail labels from candidates.
We show that our constrainedoptimization objective can be tractably solved using the Sinkhorn-Knopp algorithm [13], whichincurs only a few computational overheads.
Theoretically, we prove that our overall objective isstatistically consistent, which ensures the optimality of the learned classifier at a population level.
As an integral part of our framework, SoLar tackles the challenge of estimating the class priordistribution, which is necessitated in our constrained optimization.
Unlike conventional long-tailedlearning (LTL) scenarios [14, 15], the PLL setup is presented with label ambiguity in the proximity ofcandidate sets.
This makes it intractable to estimate the marginal class prior distribution, especiallyby counting training samples grouped by class labels.
We tackle this non-trivial problem by performing an iterative class prior estimation, which provides a strong proxy for real prior (Section 3.2).
Empirically in Figure 1 (b), we show that the estimated class prior favorably matches the ground-truthprior on a long-tailed version of the CIFAR10 dataset with partial labels.
We comprehensively evaluate SoLar on three benchmark datasets, where SoLar establishes state-ofthe-art performance.
Compared to the best baseline, SoLar improves the accuracy on tail classes by20.04 % and 23.58 % on the long-tailed versions of CIFAR10 and CIFAR100 datasets.
While our workprimarily focuses on label disambiguation from the PLL perspective, we demonstrate that SoLar canbe compatible with LTL techniques as well.
For example, we equip SoLar with logit adjustment [14],a state-of-the-art LTL method and further improve SoLar’s performance by 6.72 % on long-tailed CIFAR10.
We hope our work will inspire future works to tackle this important problem.
Problem Setup
Let X =
R d denote the input space and Y = { 1 , . . .
, L } denote the label space.
We receive a trainingdataset D
= { ( x i , S i ) } ni =1 with n examples.
Each tuple in D consists of an image x i ∈
X andthe candidate label set S i ⊂ Y .
Following previous works [6, 7], we assume the true label y i ∈
Yof x i is concealed in S i , i.e. , y i ∈ S i .
A fundamental challenge in partial-label learning is labeldisambiguation, i.e. , identifying the ground-truth label y
i from the candidate label set S i .
Our goal is to train a classifier f : X (cid:55)→ [0 , 1] L , parameterized by θ , that can perform predictions onunseen testing data.
Here f is the softmax output of a neural network, and f j ( · ) denotes the j -th entry.
Besides, we denote the prediction matrix by P =
[ p 1 , . .
. , p n ] ⊤ =
[ p ij ] n × L , where p ij = f j ( x i ) .
To perform label disambiguation, we maintain a pseudo-label matrix Q =
[ q 1 , . . . , q n ] ⊤ ∈ [0 , 1] n × Land train the classifier with the cross-entropy loss l cls ( f ; x i , q i ) =
(cid:80)
Lj =1 − q ij log( f j ( x i )) , whereq ij is the j -th entry of q i .
// Empirical class prior estimation
Proposed Method
In this section, we describe our novel S inkh o rn La bel r efinery (SoLar) framework for partial-labellearning.
As we reckon, the combinatorial challenge of partial-labeling and long-tail learning liesin matching between a decent marginal prior distribution with drawing the pseudo labels.
To copewith these problems, SoLar comprises two components.
On one hand, SoLar formulates an optimaltransport objective to facilitate the refinement of pseudo-labels to match the marginal class prior.
Theresulted constrained optimization objective encourages the model to properly draw the tail labels.
Onthe other hand, the PLL setting manifests difficulty in straightforwardly estimating the class prior.
SoLar henceforth incorporates two novel techniques—a moving-average estimation approach with acomplementary reliable set sampling mechanism.
The pseudo-code is summarized in Algorithm 1 .
3.1 Sinkhorn Pseudo-Label Refinery
Optimal Transport Objective.
We formalize an optimal transport problem for proper label assignments.
At each training step, SoLar searches for pseudo-labels Q close to the current classifier’spredictions P , while subject to some constraints:
[Equation]
r is an L -dimensional probability simplex that indicates the prior class distribution.
In other words,we would like the summation of the j -th column in Q ( i.e. , the total probability mass predicted asclass j ) to match the prior class probability.
Note that, here we temporarily assume we have a decentestimation of the class priors and we will describe the means of estimation in the following Section 3.2.
The column vector c = 1 n 1 n indicates that our n training examples are sampled uniformly.
In Eq.
(1),with a slight abuse of notation, Q is essentially a joint probability matrix post to the refinery.
Thatis, each x i is treated as a supplier holding 1 n units of goods while the j -th label needs q ij units ofgoods.
To obtain pseudo-labels, we take a re-scaled solution n Q bearing an assumption of a uniformmarginal 1 n of x i .
With further scrutiny of the objective function above, this formulation complies with two constraintswe posited, such that: (i)-it avoids biased assignment towards (only) the head labels in a long-tailed
PLL setup, with Q ⊤ 1 n = r constraining the distribution of pseudo-labeling to match the prior of the class labels.
(ii)-the probability mass of Q in each row is driven to be distributed within the candidatelabel set of a sample x i .
In Eq.
(1), we use − log( p ij ) as cost such that predictions with higher confidence enjoy a lower cost.
This can also be replaced by other cost functions h ( p ij ) as long as h is monotonically decreasing.
But, we show that the negative log-likelihood cost results in a statistically consistent risk estimator.
Theorem 1.
Denote ˜∆ the set of measures on X × Y that satisfies the constraints in Eq.
Let D pbe the distribution of tuple ( x , S ) .
The population risk of our method is as follows,
[Equation]
Let f ∗∗ and f ∗ denote the optimal classifiers trained on fully-supervised data with cross-entropy lossand PLL data with R ( f ) , respectively.
Assume the small ambiguity condition [5] is satisfied.
Then,under the deterministic scenario and ignoring the null set, we have f ∗ =
f ∗∗ .
We refer the readers to Appendix A.1 for the proof.
Theorem 1 indicates that the optimal classifier f ∗∗can be recovered by our algorithm at a population level.
Compared with other theoretically consistent
PLL risks [6, 16], our risk preserves the marginal class distribution of the estimated pseudo-labels,and is reasonably more favorable in the long-tailed learning setup.
Fast Approximation.
To resolve Eq.
(1), we adapt the well-known Sinkhorn-Knopp algorithm
[13]for efficient optimization.
Formally, we define a matrix M such that m ij =
p λij I ( j ∈ S i ) , whereλ > 0 is a smoothing regularization coefficient.
The pseudo-labels are obtained by,
[Equation]
with iteratively updated α ← c ./ ( Mβ ) ,
[Equation]
Here ./ denotes element-wise division.
α ∈ R n and β ∈ R L are known as scaling coefficientsvectors.
Detailed derivation are provided in Appendix A.2.
Here, we set m ij =
if j / ∈ S i , which isequivalent to define log(0) = − inf .
Thus, Q will also be assigned
s outside the candidate sets.
Inspired by [17], we also involve a queue acceleration trick to avoid traveling the whole training set.
We found that Eq.
(3) typically converges to a satisfactory result in < 50 rounds.
With only simplealgebra operations, this can be efficiently implemented on GPUs; see Appendix C.3.
3.2 Iterative Class Prior Estimation
Unlike conventional long-tailed learning scenarios [14, 15], the PLL setup is presented with labelambiguity in the proximity of candidate sets.
This makes it intractable to estimate the marginal classprior distribution, especially by counting training samples grouped by class labels.
In SoLar, towards estimating the class marginals, our aim is to propose a solution with (almost) fullautonomy that can adapt to the data distribution and the training state of the model.
Here, we proposetwo interdependent techniques to achieve this goal.
The resulted estimation is utilized in Eq (1).
The core of our method is two-fold: (i)-we propose a moving-average style updating rule for classprior estimation, resulting in stable training dynamics ; (ii)-we carefully devise a sample selectionmechanism for robust training, while preserving the marginal label distribution as much as possible.
Moving-Average Distribution Updating.
We propose using the model predicted results as aproxy for class prior estimation.
In spite of its simplicity, we cannot ignore the fact during the earlystage of training, the predicted labels can be largely imprecise, not trustworthy, and biased towardsthe head labels.
Therefore, we propose an update mechanism in a moving-average fashion.
Sincewe hold no assumption on the true class prior, the updating scheme is initiated from a uniform classprior r =
[1 /L, . . .
, 1 /L ] .
Respectively at each cycle (often defined by epochs), SoLar refines thedistribution as follows.
[Equation]
where
[Equation]
where µ ∈ [0 , 1] is a preset scalar.
As displayed, the class prior is progressively updated alongsidestabler training dynamics through a linear function.
As the training proceeds, the model becomesmore accurate and thus the estimated distribution becomes more reliable.
Class-wise Reliable Sample Selection.
In the early stages of training, the class prior estimationcan be inaccurate due to the poor pseudo label predictions.
Thus, we posit that the lack of propersample selection at early stages may jeopardize the marginal prior estimation.
As common as it is for
PLL learning dynamics, we want to reduce the variance of the pseudo-label quality, via a reliablesampling algorithm.
This module works together with the moving-average class prior estimation.
A brute-force but effective strategy is to employ the small-loss sampling criterion in noisy-labellearning [18], which assumes low-quality pseudo-labels can likely be characterized by a large loss.
However, in a long-tailed setup, the tail labels can also incur a large loss because of their low-shotnature, making them rarely selected.
We empirically show this phenomenon in Section 4.3.
We henceimplement the following procedures into the proposed SoLar framework.
The goal of such a moduleis to make sure the coverage of the tail labels as well as to keep the marginal prior unbiased.
To begin with, to make sure that all labels — particularly the tail labels — are coverd, we split thebatch of data B into L slices that are indexed by the argmax taken from the refined pseudo-labels:
B j = { ( x i , q i ) | j = arg max j ′ ∈ S
i q ij ′ } ⊂ B .
In what follows, we derive two combinatorial rules onthe selectivity of the training samples:
(i)
Based on
the aforementioned setup, we modify the small loss criterion in the noisy-labellearning [18] and adapt it to a
long
-tailed setup.
Specifically, we first compute the instance-wiseloss by marginalizing over the class index: l i = − (cid:80)
L j =1 q ij log( p ij ) .
Then, within the providedset B j ,
we
anchor the total sample number to be k = min( ⌈ r j ρ | B |⌉ , | B j | ) , which is associatedwith r j for distribution-preserving .
The samples are being selected with top- k smallest l i .
(ii) A parallel mechanism is inspired by the application of pseudo-labeling
schemes in semisupervised learning [19].
To adopt such approaches in SoLar, we purposely select those sampleshaving a very high confidence score which is measured by the cosine similarity: e i =
q ⊤ i p i .
As such, this second criteria selects samples that meet the inequality e i > τ .
This mechanismfurther ensures high utility on the (likely) well-disambiguated samples.
Notice that, ρ, τ > 0 are two thresholding hyper-parameters.
The two introduced mechanismsare established to work collaboratively.
Simply put, samples that satisfy either criterion are usedfor training the classifier.
We will show later in Section 4.3 the efficacy of our sample selectionmechanisms.
Experiments
In this section, we experimentally analyze the proposed SoLar method under various scenarios forthe imbalanced PLL problem.
More empirical results can be found in Appendix C.
The source codeof SoLar is attached in the Supplementary.
4.1 Setup
Datasets.
First, we evaluate SoLar on two long-tailed datasets CIFAR10-LT and CIFAR100-LTintroduced in [20, 21].
The training images are randomly removed class-wise to follow a pre-definedimbalance ratio γ = n 1 n L , where n j is the image number of the j -th class.
For convenience, classindices are sorted based on the class-wise sample size, in descending order with n 1 ≥ ... ≥ n L .
Wehave n 1 = 5 , 000 for CIFAR10-LT and n 1 = 500 for CIFAR100-LT.
We use different imbalanceratios to evaluate the performance of SoLar, with γ ∈ { 50 , 100 , 200 } for CIFAR10-LT and γ ∈{ 10 , 20 , 50 } for CIFAR100-LT.
We then generate partially labeled datasets by manually flippingnegative labels ¯ y ̸
= y to false-positive labels with probability ϕ =
P (¯ y ∈ Y | ¯ y ̸ = y ) , which followsthe settings in previous works [6, 22].
The final candidate label set is composed of the groundtruth label and the flipped false-positive labels.
We choose ϕ ∈ { 0 . 3 , 0 . 5 } for CIFAR -LT andϕ ∈ { 0 . 05 , 0 . 1 } for CIFAR100-LT.
For all experiments, we report the mean and standard deviation based on 3 independent runs (with different random seeds)
Baselines.
We compare SoLar with six state-of-the-art
partial-label learning methods: 1) PiCO [7]leverages contrastive learning to disambiguate the candidate labels by updating the pseudo-labelswith contrastive prototype labels.
2) PRODEN [6] is also a pseudo-labeling method that iterativelyupdates the latent label distribution by re-normalized classifier prediction.
3) LWS [22] also worksin a pseudo-labeling style, which weights the risk function by considering the trade-off between
[Table]
losses on candidate labels and non-candidate ones.
4) CC [16] is a classifier-consistent method thatassumes the candidate label set is uniformly sampled.
5) MSE and EXP [23] utilize mean squareerror and exponential loss as the risk estimators.
All the hyper-parameters are searched according tothe original papers.
Implementation details.
We use an 18-layer ResNet as the feature backbone.
The model is trained for1000 epochs using a standard SGD optimizer with a momentum of 0 .
The initial learning rate is setas 0 . 01 , and decays by the cosine learning rate schedule.
The batch size is 256 .
These configurationsare applied for SoLar and all baselines for fair comparisons.
For SoLar, we devise a pre-estimationtraining stage, where we run a model for 100 / 20 epochs on CIFAR10/100-LT respectively to obtaina coarse-grained class prior.
After that, we re-initialize the model weights and run with this classprior.
For our Sinkhorn-Knopp algorithm, we fix the smoothing regularization parameter as λ = 3and the length of the queue for acceleration as 64 times batch size.
The moving-average parameter µfor class prior estimation is set as 0 .
for CIFAR10/100-LT in the first stage and fixed as 0 . 01later.
For class-wise reliable sample selection, we linearly ramp up ρ from 0 .
2 to 0 .
6 in the first50 epochs and fix the high-confidence selection threshold τ as 0 . 99 .
To improve the representationability of SoLar, we further involve consistency regularization [19] and Mixup [24] techniques onreliable examples; see Appendix B.1 for more details.
For fair comparisons, we equip PRODEN withthese two techniques as well.
4.2 Main Results
SoLar achieves SOTA results.
As shown in Table 1, SoLar significantly outperforms all the rivalsby a notable margin under various settings of imbalance ratio γ and label ambiguity degree ϕ .
Specifically, on CIFAR10-LT dataset with ϕ = 0 .
5 , we improve upon the best baseline by 9.05 %,10.91 %, and 7.47 % when the imbalance ratio γ is set to 50 , 100 , and 200 respectively.
Moreover,most baselines display significant performance degradation as ϕ increases, whereas SoLar remainscompetitive.
We also verify the effectiveness of SoLar on a more challenging CIFAR100-LT datasetwith 10 times more classes and thus stronger label ambiguity.
Notably, the performance gap between
SoLar and baselines remains substantial ( > 10% in many cases).
Results on different groups of labels.
We show that SoLar achieves overall strong performance onboth head and tail classes.
To see this, we report accuracy on three groups of classes with different [Table] [Table] [Table]
sample sizes. Recall from Section 4.1 that the class indices are sorted based on the sample size, indescending order.
We divide the classes into three groups: many ( { 1 , 2 , 3 } ), medium ( { 4 , 5 , 6 , 7 } ),and few ( { 8 , 9 , 10 } ) shots for CIFAR10-LT and many ( { 1 , . .
, 33 } ), medium ( { 34 , . .
, 67 } ), andfew ( { 68 , . . .
, 100 } ) shots for CIFAR100-LT.
Table 2 shows the accuracy of different groups on both
CIFAR10-LT and CIFAR100-LT.
On few-shot classes, the gap between SoLar and the best baselineis 20.04 % and 23.58 % on CIFAR10-LT and CIFAR100-LT, respectively.
Moreover, SoLar retains acompetitive performance on many-shot labels ( i.e. , head classes).
In contrast, both PRODEN and
PiCO exhibit degenerated performance on tail classes, especially on the CIFAR100-LT dataset.
Thecomparisons highlight that SoLar poses a good disambiguation ability in the long-tailed PLL setup.
4.3 Ablation Studies
In this section, we present our main ablation analysis to show the effectiveness of SoLar.
The mainresults of our ablation studies are shown in Table 3; more results can be found in Appendix C.4.
Effect of Sinkhorn label refinery.
It is of interest to see whether the performance improvementcomes from our Sinkhorn label refinery procedure.
To see this, we equip PRODEN with our sampleselection and class prior estimation mechanism (dubbed PRODEN w S+CPE ).
As shown in Table 3,
PRODEN w S+CPE still fails to disambiguate labels on the tail, which validates the superiority ofour Sinkhorn label refinery procedure.
Effect of class prior estimation.
To verify the effectiveness of class prior estimation in SoLar, wecompare with two variants: 1) SoLar w/o Pre-Est which removes the class prior pre-estimation stageof SoLar and trains from uniformed class prior; 2) [-SoLar w Cand-Est initializes the class priordistribution by counting per-class candidate numbers ( i.e. , r j = (cid:80) ni =
 I ( j ∈ S i ) ); 3) SoLar-Oraclealso-]SoLar-Oracle also removes the pre-estimationstage but trains from the real class prior.
The results are in Table 3.
We draw two salient observations.
First, SoLar outperforms SoLar w/o Pre-Est, which suggests that using coarsely-estimated classprior is more beneficial than using uniform class prior.
Secondly, we contrast with SoLar-Oracle,which theoretically serves as an upper bound by using the real class prior. We observe that SoLar’s
[Figure] performance favorably matches or even outperforms SoLar-Oracle on few-shot classes.
The resultsare encouraging since SoLar does not require access to real class prior at all.
Effect of sample selection.
Next, we experiment with different sample selection strategies, including:1) SoLar w/o S which regards all examples as clean samples and does not perform selection; 2) SoLarw GS which performs selection in a global manner rather than class-wise; 3) SoLar w/o S-SL whichonly selects high-confidence examples.
From Table 3, we can observe that SoLar w/o S worsenedthe performance because many pseudo-labels are unreliable at the beginning.
SoLar w/o GS worksrelatively well on data-rich labels but suffers on the tail.
This happens because the majority classesdominate the sample selection.
SoLar w/o S-SL performs similarly to SoLar w/o GS on CIFAR10-LT,but diverges on CIFAR100-LT.
In the beginning, few examples are selected as the classifier is lessconfident, resulting in degenerated solution.
Our results indicate that our sample selection procedureis indispensable to SoLar in handling long-tailed class distribution.
Ablation on representation learning.
Here we ablate the contributions of two components in representation enhancement: mixup augmentation training and consistency regularization.
Specifically,we compare SoLar with three variants: 1) SoLar w/o MU which removes Mixup augmentationtraining; 2) SoLar w/o CR which removes consistency regularization; 3) SoLar w/o MU+CR whichremoves both Mixup and consistency parts; 4) PRODEN w/o MU+CR is the PRODEN algorithm thatremoves Mixup and consistency parts as well.
From Table 3, we can observe that both SoLar and PRODEN benefit from Mixup and consistency regularization techniques. In contrast to the relativelycomplicated contrastive learning modules in PiCO, we show that a simpler design can also alleviatethe representation dilemma of PLL. 4.4 Further Analysis
Combining SoLar with logit adjustment.
While our work mainly focuses on label disambiguationfrom the PLL perspective, potential improvement is straightforward by applying long-tailed learningtechniques to well-disambiguated datasets.
To this end, we equip SoLar with logit adjustment [14](dubbed SoLar-LA ), a state-of-the-art LTL technique, for improved performance.
Denote the logitsof our model by g ( · ) , i.e. , the output of the network prior to the softmax layer.
Given a testingdata x , SoLar-LA made prediction by arg max j ∈
[ L ] g j ( x ) − ζ · log( r j ) , where ζ is a temperatureparameter for calibration.
This post-hoc corrected prediction is known to be theoretically consistentfor minimizing the balanced error [14].
Note that logit adjustment cannot be applied to the original
PLL data directly, as it requires the true class prior as well as a classifier trained by minimizing thelong-tailed empirical risk.
Fortunately, SoLar achieves both goals.
Hence, we implement SoLarLA by using the classifier as well as the estimated class prior of SoLar.
Figure 2 (a) reports theperformance of SoLar-LA with varying ζ values.
With a proper ζ , SoLar-LA improves upon SoLarby 6.72 % on CIFAR10-LT.
This suggests that SoLar is compatible with existing LTL methods andopens the door to exploring more advanced LTL techniques.
Results on fine-grained partial-label learning.
In practice, semantically similar classes can lead tosignificant label ambiguity, as exemplified in Figure 1 (a).
To test the limit of SoLar, we follow PiCO[7] and evaluate on two fine-grained datasets: 1) CUB200-LT [25] dataset with 200 bird species; 2)
CIFAR100-LT with hierarchical labels (CIFAR100-H-LT), where the candidate labels are generated within the same superclass 1 .
We set ϕ = 0 .
05 , γ = 5 for CUB200-LT and ϕ = 0 .
5 , γ = 20 for
CIFAR100-H-LT.
In Figure 2 (b) and (c), we compare SoLar with two strongest baselines PRODENand PiCO, where SoLar improves the best baselines by a large margin ( 4.37 % on CUB200-LT and3.67 % on CIFAR100-H-LT).
These results clearly validate the effectiveness of SoLar, even when thedataset presents severe label ambiguity.
Related Work
Partial-label Learning (PLL) [4] allows each example to be equipped with a candidate label setwith the ground-truth label being concealed.
A plethora of PLL methods [5, 26, 27, 28] have beendeveloped.
For example, maximum margin-based methods [29, 30] maximize the margins betweencandidate labels and the remaining ones.
Graph-based approaches [31, 8, 9, 10] typically leveragenearest neighbors to smooth the logical label vectors.
Recently, pseudo-labeling methods [16, 6,22, 32, 7, 33] become increasingly prevalent due to their promising performance, e.g. , PRODEN[6] re-normalizes classifier’s outputs and PiCO [7] updates pseudo-targets by using contrastivelylearned prototypes.
However, most of these works overlook the fact that class distribution can beimbalanced.
Our empirical studies demonstrate that state-of-the-art PLL methods fail to disambiguatetrue labels on the minority classes, which motivates us to explore the optimal transport-based labeldisambiguation technique for PLL.
Long-tailed Learning (LTL) assumes that the training data follow Zipf’s law that few labels posea large number of instances, which is observed in a wide range of applications [11].
To cope withthe LTL problem, sampling-based methods [34, 35] re-balance different classes at a data level, e.g. ,up-sampling on minority classes.
Re-weighting methods [20, 14, 36, 37, 15] typically adjust theweights of instances or classes that calibrates the classifier predictions to be balanced.
Some worksalso explore transfer learning [38, 39] or ensemble learning [40, 41] in LTL.
A recent trend in LTL[42, 43] is to decompose representation and classifier learning.
Nevertheless, these methods assume afull-supervision of training data, which is unrealistic in many real-world applications.
It has motivatedresearchers to study the LTL problem in weak-supervision scenarios, such as semi-supervised learning[44, 21] and noisy-label learning [45, 46].
In this work, we explore a different formulation of LTLwhere only candidate labels are provided.
Optimal Transport (OT) [12] is originally proposed to measure the distance between two measures.
Recently, it has drawn huge attention from different fields in machine learning, including semisupervised learning [47], domain adaptation [48, 49], object detection [50], and generative models[51].
The most related one to our work is OTLM [15], which also applies OT to the supervised LTLproblem.
By assuming a known test class distribution, OTLM performs OT-based post-hoc correctionon model predictions to obtain balanced results.
In contrast, our work targets the long-tailed PLLproblem and holds no assumption on class priors.
Conclusion
In this work, we present a novel Sinkhorn label refinery framework (SoLar) for a challengingimbalanced partial-label learning problem.
Key to our method, we derive an optimal transportobjective that refines the pseudo-labels to match the true class prior.
Additionally, we tackle thenon-trivial class prior estimation problem by a moving-average technique, which is further guardedby a sample selection mechanism for controllable model training.
Empirically, SoLar can run withoutknowledge of the real class distribution.
Comprehensive experiments show that SoLar improvesbaseline algorithms by a significant margin.
As we collect the real-world applications, the partiallabeled tasks often arise together with a long-tailed class distribution.
That is, from the perspectiveof human annotators, a long-tailed label is much harder to tag and that may subsequently lead to apartial-labeled candidate set.
In the future, we hope to explore this line of research in broader tasks.
We hope our work will draw more attention from the community towards a broader view of tacklingthe imbalanced partial-label learning problem.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen- tal materials.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Appendix C.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
AppendixA Theoretical ProofsA.1 Proof of Theorem 1
First, we provide the following lemma to show the consistency of the standard cross-entropy loss.
Lemma 1.
If the cross-entropy loss is used as loss function, the optimal classifier f ∗∗ that minimizesthe population risk R CE ( f ) =
E x ,y [ − log( f y ( x ))] satisfies f ∗∗ y ( x )
=
p ( y = i | x ) .
Proof.
We provide a proof sketch in the sequel, and a similar result has been shown in [52, 16].
Thecross-entropy loss leads to the following optimization problem,
[Equation]
By introducing a multiplier ξ , the corresponding Lagrangian is as follows,
[Equation]
Setting the derivative to 0 yields that,
[Equation]
We conclude that ξ = 1 and f ∗∗ y ( x ) = p ( y = i | x ) holds.
Next, we provide the definition of the (theoretical) ambiguity degree [5] to establish the learnabilityof the PLL problem.
Definition 1.
Denote the distribution of triplet ( x , y, S ) by ˆ D p , We define the ambiguity degree as,
[Equation]
We say a data distribution satisfies the small ambiguity degree condition if κ < 1 .
This is a naturalrequirement.
If κ = 1 , there exists at least one label pair that always co-occur, and thus, it isimpossible to find the optimal hypothesis given partial-labeled data.
Now, we provide the main proof sketch for Theorem 1.
Note that we always seek an optimaljoint probability matrix before model training, which is mainly designed for empirical measuresof the data samples.
At a population level, we aim to search for an optimal probability measure that meets the marginal constraints and candidate constraints.
Note that we aim to minimize
E ( Q , P ) =
KL ( Q || P ) + H ( Q ) where KL ( · ) is the Kullback–Leibler divergence and H ( · ) is anentropy regularizer.
This gives rise to our population risk,
[Equation]
Here we omit the entropy term H ( ϖ ) as it serves as a regularizer.
As the training labels arecategorical, we may assume H ( p ( y | x ))
= 0 , and thus, the infimum still holds.
Otherwise, we canoffset it by setting the smoothing parameter λ = 1 in the Sinkhorn-Knopp approximation.
When thepseudo-labels are fixed, this objective is exactly the cross-entropy loss whose target measure is ϖ .
To show the consistency, we first prove that f ∗∗ is the minimizer of R ( f ) .
Substituting f ∗∗ into
R ( f ) and combining with Lemma 1, we obtain that,
[Equation]
It is obvious that ϖ = p ( y | x ) leads to the minimal KL-divergence.
On the other hand, we show that f ∗∗ is the unique solution.
We assume there exists another hypothesisf ′
that minimizes Eq.
(9) and holds a different prediction
y ′ ̸
= y from f on at least one instance( x , y, S ) .
By the deterministic assumption, we have R ( f ′ )
= 0 , which holds only if y ′ is invariablyincluded in the candidate set S , i.e. , P ˆ D p ( y ′ ∈ S | x , y )
= 1 .
Clearly, it violates the small ambiguitydegree condition and causes a contradiction.
Ignoring all the null set where p ( x , y ) = 0 , we concludethat the minimizer f ∗ of our population risk equals the fully-supervised f ∗∗ .
The above discussion indicates that our method poses good performance guarantees like existing
PLL methods [6, 16].
A similar loss R ( f ) =
E ( x ,S )
∼D p min y ∈ S − log( f y ( x )) is also discussed in[6], which is known as minimal loss .
Its main difference from our objective is that we seek the bestprobability measure instead of an example-wise minimum.
Our risk preserves the marginal constraintof the estimated pseudo-labels and is reasonably more favorable in the long-tailed learning setup.
A.2 Derivation of the Sinkhorn-Knopp Iteration
In this section, we briefly introduce the derivation of the Sinkhorn-Knopp algorithm to ensure theintegrity of our work.
Recall that Eq.
(1) is a standard linear programming (LP) problem, and can be solved in polynomialtime.
But, considering the high volume of data points as well as potential large class space, common
LP solvers typically become time-consuming.
To this end, we investigate a smoothed version ofthis optimization problem for fast approximation.
Technically, we further add a negative entropyregularization term to obtain the following objective,
[Equation]
This entropy regularizer is derived from an optimization perspective and is different from the one inour derivation of E ( Q , P ) =
KL ( Q || P ) + H ( Q ) .
The resulting objective becomes smoothing andconvex, and thus, can be efficiently solved.
It is worth pointing out that in the PLL problem, we have a special constraint that no probability massis assigned outside the candidate label sets.
To avoid introducing another set of annoying Lagrangemultipliers, we transform this constraint to infinity costs on non-candidate labels.
That is, we define acost matrix T such that T ij = − log( p ij )
I ( j ∈ S i ) + inf I ( j / ∈ S i ) and here we assume 0 · inf = 0 .
Now, we obtain the Lagrangian of the new optimization problem,
[Equation]
where u , v are Lagrange multipliers.
As the original optimization is convex, the solution has tosatisfy the Karush-Khun-Tucker (KKT) conditions.
Therefore, we have the following equations,
[Equation]
Let M be a matrix such that m ij =
e − λT ij = p λ ij I ( j ∈ S i ) .
We can get that,
[Equation]
Thereby, solving the primal problem in Eq.
(11) equals to finding the multipliers u and v .
Again,this is equivalent to get another two vectors α ∈ R n , β ∈ R L such that α i =
e − 12 − λu
i andβ j = e − 12 − λv j , which are also known as scaling coefficients vectors.
Then, we can get the followingequation,
[Equation]
Recall that we enforce Q to meet the following constraints,
[Equation]
This gives rise to an alternative coordinate descent algorithm for updating the scaling coefficients,
[Equation]
It is also known as the Sinkhorn-Knopp fixed point iteration.
We refer the readers to [53] for itsconvergence properties.
Empirically, we find that setting λ = 3 and running 50 steps are enough toget a satisfactory solution.
The final step is to take a re-scaled the minimizer n Q in Eq.
15, since Qserves as a joint probability matrix
and the posterior is calculated by p ( x ,
y ) p ( x ) =
np ( x , y ) .
Without lossof generality, we slightly abuse the notation Q to denote the obtained pseudo-labels.
The core of our algorithm is to replace our zero constraints on non-candidates with infinity cost,which equals defining log(0)
= − inf .
These infinity values are then mapped back to 0 s in the
Sinkhorn-Knopp iteration, ensuring the feasibility of the calculation.
B Practical Implementation
In this section, we describe several details of the practical implementation of SoLar.
B.1 Details of Representation Enhancement
A recent work PiCO [7] has shown that existing PLL methods are typically trapped in adisambiguation-representation dilemma , where the low-quality representation and imperfect label disambiguation mutually deteriorate each other.
It can be even worse in the imbalanced setupbecause of low-shot examples on the tail.
While PiCO pioneers the contrastive learning technique in
PLL for superior performance, we show this can be achieved by a much simpler design.
In contrast tothe complicated network architecture in PiCO [7], we involve consistency training regularizer alongwith mixup augmentation to improve the representation quality.
Consistency Regularization.
Recently, consistency regularization (CR) has been widely appliedin weakly-supervised learning [19], which assumes that a classifier should produce similar classprobability for a sample and its local augmented copies.
Motivated by this, we also incorporated
CR into PLL.
Given an image x i , we adopt two data augmentation modules SimAugment [54] and
RandAugment [55] to obtain a weakly augmented image x wi and strongly augmented counterpartx si .
During training, the weak one is utilized to produce pseudo-labels by our Sinkhorn labelrefinery procedure as well as selecting the reliable subset.
Then, we define the the CR loss l cr =− (cid:80)
Lj =
q ij log( f j ( x si ))
that is the cross-entropy loss on x si and q i .
MixUp.
We further incorporate mixup training for improved performance.
Given a pair of weaklyaugmented examples x wi and x wj in the reliable set, we create a virtual training example by linearlyinterpolating both,
[Equation]
18) where σ ∼ Beta ( ς, ς ) and we simply set ς = 4 without further tuning.
Similarly, we define the mixuploss l mix as the cross-entropy loss on x mi and q mi .
In our implementation, we add CR and MixUP on reliable examples only.
For the remaining examples,since their pseudo-labels are unreliable, we train them with the classical re-normalized PLL loss l rn[6].
The final loss is defined by,
[Equation]
(19) where l ce is the original classification loss of weakly-augmented instances.
We linearly ramp up ηfrom 0 to 0 .
9 in the first 50 epochs, which helps warm up the classifiers.
B.2 Relaxed Solution for the Sinkhorn-Knopp Algorithm
Recall that we start from a uniform class prior.
Empirically, most of our experiments run very well.
But, in some cases, it can result in an unsolvable optimal transport objective.
For example, if thereare only a few examples holding one specific label as a candidate ( ≪ n L ) , we have no hope toconstrain the sum of pseudo-labels on this label to be the average number of instances.
Thus, the
Sinkhorn-Knopp iteration diverges.
In our implementation, once divergence occurs, we return a relaxed solution to the optimal transportproblem.
Concretely, we modify the matrix M by adding a small number ϵ
= 1 e − 5 on its zero
[Table]
entries.
In other words, we regard all negative labels as potential candidates but assign them verylarge costs.
Now, M is an element-wise positive matrix, and the Sinkhorn fixed iteration processguarantees to converge.
We can re-run the Sinkhorn-Knopp iteration to obtain a relaxed solution.
Finally, we set the non-candidate labels as zero again, as they should not be assigned any probabilitymass.
By then, our algorithm can safely run with a uniform distribution.
This procedure is easy toimplement and increases only 1 × computation at most.
Moreover, as the class prior is estimatedbetter and better, we typically do not need this relaxed solution anymore.
In practice, it is not likely that we are fully unknowledgeable of the class distribution.
Thus, we canalso use a good initialization to avoid the aforesaid problem.
Empirically, we find that initializingwith the ratio of candidate label number is also a good choice for SoLar.
C Additional Experimental Results
In this section, we report the additional empirical results of our proposed SoLar framework.
Allexperiments are conducted on a workstation with 8 NVIDIA A6000 GPUs.
The licenses of ouremployed datasets are unknown (non-commercial).
C.1 Experimental Setup on Fine-Grained Datasets
In the sequel, we show the full experimental setups on fine-grained classification datasets.
Inparticular, on CUB200-LT, we set the batch size as 128, and the length of the queue for Sinkhornacceleration as 8 times batch size.
We train the model for 500 epochs without the pre-estimationtraining stage.
Other hyper-parameters are the same as our default setting.
On CIFAR100-H-LT, wesimply adopt the default parameter configurations.
The baselines are also fine-tuned to achieve theirbest results.
C.2 Full Results of SoLar with Logit Adjustment
In Table 4, we report the full results of SoLar-LA with varying ζ values on CIFAR10-LT and
CIFAR100-LT.
On CIFAR100-LT, SoLar-LA still outperforms SoLar with a proper ζ .
The resultsdemonstrate that SoLar achieves promising disambiguation ability in the imbalanced PLL setup.
Given well-disambiguated data, SoLar makes it possible to apply off-the-shelf LTL methods forfurther improvements.
C.3 Running Time of the Sinkhorn-Knopp Algorithm
[Table]
As we mentioned in Section 3.1, our SinkhornKnopp algorithm can be efficiently implementedon GPUs.
In Table 5, we report the total running time ( 1000 epochs) of the Sinkhorn-Knoppiterations as well as model training based on ourimplementation.
We run SoLar with our defaultparameter configurations and evaluate using one
NVIDIA A6000 GPU.
During training, we maintain a queue of size 64 × 256 to store classifier predictions in previous 64 steps.
Then, we concatenatethe prediction in the current batch with the queue to run the Sinkhorn-Knopp algorithm.
It can be [Figure] shown that the Sinkhorn-Knopp iterations take less than 1 / 10 time cost than regular model training(including forward pass and backward propagation).
These results clearly validate the efficiency ofour algorithm.
C.4 More Ablation
Effect of selection threshold ρ and τ .
We further investigate the effect of the small-loss selectionratio parameter ρ and the high-confidence threshold τ .
Figure 3 (a) shows the performance of
SoLar with varying ρ on CIFAR10-LT (without ramp-up).
When ρ = 0 , SoLar simply selectshigh-confidence samples, which leads to an unsatisfactory performance on few-shot labels.
Theperformance becomes much better as ρ becomes larger and achieves the best when ρ = 0 .
But,when ρ becomes too large, the model tends to overfit unreliable labels.
Empirically, we find that
SoLar works well in a wide range of ρ and ρ ≈ 0 .
5 is a good choice.
[Table]
Table 6 lists the result of SoLar with varying τ values.
With a relatively small τ , SoLar collects too manyunconfident examples, which typically hurts the robustness of model training.
When τ = 0 .
99 , SoLarachieves rather promising results on different groupsof labels.
A surprising observation is SoLar without high-confidence selection, i.e. , τ = 1 , achievesslightly better results than SoLar on few-shot labels.
We find the reason is that those well-disambiguatedexamples on the majority classes are overlooked, which poses an effect of down-sampling and benefitslearning on the minority classes.
As a negative effect, this sacrifices the performance of many-shotlabels, which leads to an interesting trade-off.
We empirically set τ = 0 .
99 to retain relatively goodperformance on many-shot labels.
[Table]
The role of unreliable examples.
In our implementation, we enable unreliable examples to be trained withre-normalized PLL loss since their pseudo-labels canbe noisy.
This not only improves data utility but alsoserves as a warm-up mechanism for SoLar as we aimto train SoLar without knowledge of the true classprior.
To see the role of unreliable examples, we alsoevaluate SoLar without l rn and the results are shown in Table 7.
We observe that SoLar w/o l rn obtainsfavorable results on tail labels, even trained with only reliable examples.
This verifies the importanceof our distribution-preserving sample selection mechanism.
Given the oracle class prior, SoLar w/ol rn obtains competitive performance and matches our main results.
In practice, proper utilization ofunreliable samples typically leads to more promising results.
C.5 Results with Non-Uniform Data Generation
In reality, some labels may be more analogous to the ground-truth than others, and CIFAR100-H-LT isexactly one of the cases.
Following [7, 22], we further test SoLar with a non-uniform data generation process, with the following flipping matrix:
 0 .
· · ·  (20) where each entry denotes the probability of a label being a candidate.
From Figure 3 (b), weevaluate SoLar on CIFAR10-LT with an imbalance ratio γ = 100 .
It can be observed that SoLar stilloutperforms the baselines, which further validates its strong disambiguation ability.
D Societal Impact
In this section, we briefly discuss several societal impacts of our work.
The most obvious meritof our study is to reduce the cost of annotation by enabling coarse-grained labeling.
This is atwo-edged sword for the community.
On the one hand, non-expert annotation can be employed forcrowdsourcing labeling.
On the other hand, if the partial-label learning paradigm is widely applied,the need for precisely annotated data would be significantly reduced, which may cause potentialemployment destruction as a consequence of reducing the need for human labelers.
Another potentialapplication of our method is data privacy.
For instance, we may ask respondents to answer someprivate information when collecting some survey data.
The candidate set-style labeling enables therespondents to exclude several wrong answers, which would be more privacy-friendly.